Compare commits

...

219 Commits

Author SHA1 Message Date
overtrue 6de965ae3c test(e2e): register paginator regression in Darwin inventory 2026-10-03 07:07:41 +08:00
overtrue 7c56853067 fix(storage): publish delete rollback backups atomically
Stage rollback metadata outside the rollback directory and publish it only after the full write succeeds. A short write must not leave a backup that quorum rollback can rename over acknowledged version history.

Add an isolated real short-write regression and register the backported ListParts SDK test in the smoke and Linux full inventories.
2026-10-03 06:43:25 +08:00
Hauser 6f61761c7b fix(s3): honor sparse ListParts markers and verify termination
Resume part listings at the first part above the numeric marker, even
when that marker is absent. Use binary search over the sorted part
numbers and retain the existing exact-tail empty-slice path.

Add storage, XML, and real AWS SDK paginator regressions for empty and
terminal pages, sparse markers, and multipart completion integrity.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 673031eea1)
2026-10-03 06:32:24 +08:00
Chapman 1bae764530 fix: emit NextPartNumberMarker only when ListParts is truncated
ListPartsInfo.next_part_number_marker was a non-optional usize that
defaulted to 0 and was only assigned when the response was truncated.
The S3 serializer then emitted it unconditionally as Some(0), causing
AWS SDK paginators to loop infinitely on part_number_marker=0 instead
of terminating.

Change the field to Option<usize> (None by default) and set it only
inside the is_truncated branch. The S3 output layer now uses
.and_then() so NextPartNumberMarker is absent when IsTruncated=false,
matching AWS S3 behavior.

Fixes #8208

(cherry picked from commit 44de803a38)
2026-10-03 06:32:24 +08:00
overtrue 7150ed2066 ci: remove Docker Hub description sync 2026-10-02 19:59:28 +08:00
overtrue c04d566b26 fix(heal): wait for held legacy responsibility in replay test 2026-10-02 03:13:15 +08:00
Hauser 4dd267940c fix(scanner): diagnose and verify pause backlog recovery (#8293)
* fix(scanner): diagnose and verify pause backlog recovery

Expose the retained replica snapshot and claimed membership in abnormal
admin status responses. Keep diagnostics off metrics updates and verify
single-pool recovery and conflicting-proof preservation across 24 sets.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(scanner): move replica snapshots into diagnostics

Consume the terminal admin read snapshot in a single state match and move
membership, revision, and error buffers into the response. Verify buffer
handoff and the unchanged JSON contract without altering ledger authority.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-02 01:09:00 +08:00
Hauser 2aad57c984 chore: merge upstream release branch
Integrate the updated release/1.0.1 branch before pushing the
dependency and atomic API maintenance changes. Preserve the upstream
removal of async-recursion.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-10-02 01:00:04 +08:00
Hauser dc79582e40 improve 2026-10-02 00:54:32 +08:00
Hauser 25fb3d2516 chore: refresh dependencies and atomic update APIs
Update workspace dependencies and the lockfile. Replace deprecated
atomic fetch_update aliases with try_update while preserving closures
and memory ordering.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-10-02 00:46:57 +08:00
Hauser 5c39ed1495 fix(s3): queue bucket operations and restore strict Clippy checks (#8290)
* fix(s3): queue concurrent bucket creation and deletion

Keep eight active bucket transactions and bound admission waiting to 128 requests and 30 seconds. Preserve detached transaction ownership and return Retry-After with overload responses.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): restore strict Clippy compatibility on Rust 1.99

Use try_update without changing atomic ordering or overflow behavior. Keep
recursive storage futures boxed once at each frame and remove the redundant
async-recursion macro, including its non-recursive SQL planner use. Remove
needless closure borrows and orphaned dependency entries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-02 00:41:05 +08:00
overtrue da31b53dd6 fix(scanner): pass fence store to checkpoint fixture 2026-10-01 19:52:48 +08:00
Hauser a6bee557e9 fix: add UploadPart OOM validation guardrails (#8287)
* docs(release): validate candidates on release branch

* fix(scanner): validate checkpoints against global cycle fence (#8278)

## Related Issues

Related to rustfs/backlog#2701.

## Summary of Changes

Route scanner checkpoint cycle and leader validation through the global store while retaining the owning set for cache persistence, CAS revisions and publication admission.

## Verification

Two independent final-diff source reviews found no issues across correctness, concurrency and durability, test coverage, compatibility, performance and simplicity on head `8b8fe51d092090b053f552ae283960e2e306be33`. Root approval `5373624714` is bound to that exact head. Regression tests cover real two-pool routing, stale fences, post-save rejection and CAS conflicts; their reported local execution belongs to the PR author, not this merge operation. Current required CI remains pending, and this authorized admin squash does not establish CI or runtime acceptance.

## Impact

Restores checkpoint progress when global cycle and leader state differ from a set-scoped view. No format, retry, timeout, assertion or scanner-policy changes are introduced by this diff. The three prior main scanner failures remain unproved repaired.

## Additional Notes

Full validation must run on the resulting exact main revision. Reverting this patch restores the earlier set-scoped fence lookup and its checkpoint rejection behavior.

* fix(ci): restore E2E membership and pagination timeouts (#8281)

* ci: locate the auto-testing checkout for lanes that run evidence from a subdirectory (#8279)

## Related Issues

Follow-up to #8229.

## Summary of Changes

Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence.

## Verification

The exact PR head b66129ab9f passed one mechanical correctness and simplicity review, nine real-Git layout and provenance checks, and sixteen existing evidence/envelope tests. The baseline sibling layout failed with git exit 128; the corrected layout succeeded while revision mismatches and missing checkouts stayed rejected. Current PR checks are completed with successful or skipped conclusions, including the aggregate.

## Impact

Both lane and private-script revision checks remain intact. No time limits, assertions, production behavior, or evidence validation requirements change. The synthetic layout checks do not execute the actual scheduled security suite; integrated main CI and release acceptance remain separate gates.

## Additional Notes

Approved review 5374559393 is bound to the exact head above. Reverting the single-file change restores the previous checkout lookup.

* fix: add UploadPart OOM validation guardrails

Add a Docker validation harness for backlog#2704 so the ordinary UploadPart
low-concurrency memory workload can be reproduced with comparable case metadata,
process/cgroup sampling, TLS and metrics toggles, cache-env controls, and write
reclaim/direct-write experiments.

Warn when operators set the unrecognized RUSTFS_OBJECT_CACHE_* variables that
appeared in the reporter compose file. The variables are reported but remain
ignored, so startup does not silently change object data cache behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: AL <allan.bednarowski@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-01 19:32:52 +08:00
Hauser 7b76c655a9 Gate multipart copy through write admission (#8283)
Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-01 19:27:07 +08:00
Hauser 09a5c7b372 [release/1.0.1] Gate multipart copy through write admission (#8284)
Gate multipart copy through write admission

Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-01 16:06:03 +08:00
overtrue f37df93df5 fix(test): await HTTP sender readiness in Top RPC fixture 2026-10-01 09:51:54 +08:00
overtrue 4816453cd5 fix(usage): combine identical snapshot retention branches 2026-10-01 08:06:04 +08:00
Hauser f715e88f8a fix(obs): distinguish allocator counters from live memory (#8274)
* fix(obs): distinguish allocator counters from live memory

Preserve count/counter semantics and mark requested-byte attribution unavailable when live statistics or sampling are missing. Document sustained multipart memory diagnosis.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(obs): parse allocator statistics from one node

Resolve each statistic before interpreting its shape, avoiding unsupported-field tree scans and mixing data across wrapper scopes. Preserve unavailable-statistics policy and add precedence regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): isolate late parity recovery from metadata hedges

Use the existing object-scoped hedge timer barrier in exact-count recovery fixtures. Preserve payload and total-read assertions and verify that the omitted parity disk is read only during late refresh.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-01 07:57:22 +08:00
Hauser 6c06b4fe16 test(scanner): verify checkpoint takeover and repair dispatch (#8275)
* test(scanner): cover checkpoint handoff and repair dispatch

Drive runtime budget expiry, partial-cycle persistence, leadership claims,
and stale checkpoint rejection between real disk-backed fixture scans.
Verify that a metadata repair beyond the first bounded prefix is saved in
the scanner ledger and dispatched by the MRF consumer.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* ci: isolate scanner fixtures and refresh full e2e membership

Reserve nextest capacity for the real-disk scanner publication and MRF
admission fixtures. Bind both platform membership checks to the reviewed
pagination deadline test added on main.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(scanner): consolidate checkpoint fixture lifecycle

Keep one durable control store across timeout and leadership transitions,
and inject generation advancement into the shared checkpoint scenario.
Check the actual saved metadata path so late-write rejection also proves
that existing checkpoint bytes remain intact.

Centralize MRF fixture isolation and reuse nextest process isolation when
the startup environment already satisfies the test contract.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-01 07:28:37 +08:00
Hauser 49462ae5d4 fix(usage): recover historical counters with generation fencing (#8273)
* fix(usage): recover historical counters during continued writes

Use newer converged scanner snapshots to reconcile stale absolute usage
baselines while preserving concurrent mutation and expiry receipt fences.
Cover durable publication, admin and quota reads, and legacy generations.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(usage): fence snapshots and move preserved cache entries

Apply the cached scanner generation floor before every reconciliation path
and retain it even when an older snapshot happens to match core counts.
Move preserved usage entries instead of cloning their histogram maps under
the cache lock, retaining expiry receipt identity and cancellation fences.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-01 06:32:34 +08:00
overtrue 4b50b98d22 fix(ci): locate security chain scripts in the workspace 2026-10-01 06:12:54 +08:00
overtrue bcf83d4e41 fix(test): supply CPU workload for sampler regression 2026-10-01 04:59:54 +08:00
overtrue 65ce936131 test(e2e): retain startup and shutdown failure diagnostics 2026-10-01 01:46:41 +08:00
Peder Bergan e4d1ade702 fix(usage): reconcile stale counters after lifecycle expiration (#8108)
* fix(usage): reconcile stale counters after lifecycle expiration

* fix(usage): account lifecycle expiry during continuous writes

* test(usage): run lifecycle usage scenarios on one scanner store

* test(usage): use a Windows-representable pre-mutation offset

* fix(usage): harden expiry accounting recovery and quota checks

Borrow expiry receipt bucket names and avoid allocating a map key on cache hits. Cover cancelled receipts, durable snapshot recovery, and legacy quota admission after scanner confirmation. Use representable timestamp offsets in the quota regression.

Refs rustfs/backlog#2689

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(usage): recover stale persisted counts through the scanner

Seed incorrect complete usage for empty and retained-object buckets, then run the real scanner and publication consumer without further object mutations. Verify durable and admin usage over two cycles instead of writing a corrected snapshot in the test.

Refs rustfs/backlog#2689
Refs rustfs/backlog#2691

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): bound delimiter pagination fixture concurrency

The 120-second smoke timeout expired after 1018 of 1200 serial fixture PUTs, before LIST ran. Prepare the same objects with at most eight concurrent requests and await every PUT. Retain the timeout and strengthen exact prefix, KeyCount, empty Contents, and continuation-token assertions with phase diagnostics.

Refs rustfs/backlog#2689

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(upgrade): establish a persisted previous-release baseline

Seed the pinned previous-release cluster and restart it once with its data intact before replacing any node. Require every old writer to pass the strict readiness probe and preserve the seed through both mixed phases and the final current cluster. Keep InternalError fail-fast behavior and all existing compatibility deadlines and assertions.

Refs rustfs/backlog#2689
Refs rustfs/backlog#2384

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): bound cancellation metadata persistence waits

Use the system-bucket incarnation boundary now supplied by main PR #8268. Bound the three cancellation waits that previously hung during pool.bin persistence, retaining their remote-generation, target-cohort, and durable-state assertions.

Refs rustfs/backlog#2697
Refs rustfs/backlog#2689

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-30 22:33:00 +08:00
Hauser 86fc35b49f fix(scanner): expose pause backlog replica diagnostics (#8258)
* fix(scanner): expose pause backlog replica diagnostics

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(connect): stabilize runtime profile lease cancellation

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(connect): tolerate delayed schedule startup in CI

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): retry quota reads during usage warmup

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): avoid meta-bucket incarnation self-deadlock

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(test): use persisted incarnation in heal fixture

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-30 22:25:31 +08:00
overtrue b54b32b8a9 fix(deps): replace yanked yoke-derive release 2026-09-30 22:24:45 +08:00
overtrue aff1c025af fix(ci): include pagination regression in full E2E selection 2026-09-30 22:03:46 +08:00
Chris 2806a80c91 fix: prevent metadata lock reentry and stabilize CI fixtures (#8257)
* fix(test): establish a writable previous-release upgrade baseline

* fix(ci): reserve capacity for durable admin fixtures

* fix(ci): group durable IAM state fixtures by resource needs

* ci: run E2E doctests with the E2E dependency graph

* fix(ci): separate fixture startup from transport deadlines

* fix(ci): bound pagination after seeding and revisit restored copies

* fix(ci): make filesystem fixture timing deterministic

* fix(ecstore): avoid metadata lock reentry during internal mutations

* test: align recovery fixtures with durable ownership contracts
2026-09-30 20:57:07 +08:00
Chris 8655c38f1d fix(rio): preserve bounded retries after peer EOF (#8272) 2026-09-30 20:18:28 +08:00
Chris 88d03d8199 fix(storage): prevent metadata deadlocks and abandoned writes (#8268)
## Related Issues

N/A

## Summary of Changes

Reject system-bucket incarnation lookups before entering the pool metadata owner. Keep each selected scanner backlog publication cohort in one task so waiter cancellation cannot abandon its remaining serialized conditional writes. Bind repair and replay fixtures to persisted bucket incarnations.

## Verification

Head f6d44603fc received an approval from houseme in review 5365588011. This merge does not add a local runtime validation claim. Full main CI remains a separate publication gate.

## Impact

System metadata writes avoid recursive pool locking. Publication retains per-replica conditional writes and stops a cancelled caller from advancing to its next publication phase. Fixture changes supply the identities required by existing repair admission rules.

## Additional Notes

Reverting this change restores the previous behavior.
2026-09-30 19:49:24 +08:00
Hauser eb350ad20d fix(ecstore): purge empty recursive bucket orphans (#8261)
## Related Issues

Related to rustfs/backlog#2688.

## Summary of Changes

Reclaim metadata-less orphan directory trees after a complete, first-page, empty recursive root listing in an authoritative never-versioned bucket. Both listing implementations use the same fail-closed cleanup decision.

## Verification

Head b4a9120173 received an approval from loverustfs in review 5365460152. This merge does not add a local runtime validation claim. Full main CI remains a separate publication gate.

## Impact

Empty recursive root listings can remove otherwise unaddressable physical residue. Versioned buckets and populated listings remain excluded; unreadable disks, object metadata, unknown files, and uncommitted data prevent cleanup.

## Additional Notes

Reverting this change restores the previous orphan-directory behavior.
2026-09-30 19:24:32 +08:00
KarlEmm e573fb087b ci(helm): sync README to root of helm repository on release (#8263)
Include helm/rustfs/README.md in the uploaded helm-package artifact so
publish-helm-package updates the root README.md in rustfs/helm.

The build-helm-package job copies helm/README.md into helm/rustfs/
before running helm package, but upload-artifact previously uploaded
only helm/rustfs/*.tgz. Because both files share the helm/rustfs/
parent directory, actions/upload-artifact places both the chart
archive and README.md at the artifact root, and download-artifact
extracts README.md into the root of rustfs/helm alongside the tarball.
2026-09-30 15:57:24 +08:00
Chris 099a408d8b test(ecstore): isolate dispatch shutdown recovery fixtures (#8265) 2026-09-30 15:56:48 +08:00
Chris 5c3a2ff277 fix(ci): reserve capacity for profile cancellation probe (#8259) 2026-09-30 15:46:20 +08:00
Chris bb04355a3d fix(heal): retry contended local metadata CAS (#8267) 2026-09-30 15:45:59 +08:00
Chris fab2d7e144 fix(scanner): serialize pause backlog replica publication (#8264) 2026-09-30 15:45:37 +08:00
Hauser d60dfbb826 fix(heal): finalize durable MRF legacy responsibility lifecycle (#8254)
## Related Issues

Resolves the corrected lifecycle fixture findings in PR #8254.

## Summary of Changes

Preserve separate incarnation-bound durable MRF responsibilities and their lifecycle audit records, and align replay fixtures with their checkpoint identities.

## Verification

Two independent source reviews and changed-delta reviews are complete. The final review approved 607a0b9bb0 with no remaining supported findings. Local runtime claims were not independently reproduced for this pull request.

## Impact

Keeps storage generation fences and retained responsibility semantics. Test-capacity reservations preserve existing deadlines and assertions.

## Additional Notes

Squash merge of the currently approved fix under the authorized CI-bypass exception. Main CI and release acceptance remain required.
2026-09-30 14:02:32 +08:00
Chris 6bc2e10bc1 fix(tier): keep recovery worker after startup reload failure (#8260) 2026-09-30 12:05:26 +08:00
Chris 3268c42e00 fix(ci): stabilize registration and rolling upgrade readiness (#8244)
## Related Issues

Follow-up to #8233.

## Summary of Changes

Registration runtime fixtures timed out in the workspace CI lane while the same cases passed in the feature lanes. Reserve nextest capacity for the exact `connect_registration` binary, as already done for related inventory and drive fixtures. Keep its existing deadlines, internal concurrency, assertions and zero-retry policy; report the last watch status on failure.

Rolling upgrades could pass `ListBuckets` readiness while restarted peers still lacked write quorum. Before each mixed-version phase, probe writes through every node outside the asserted workload prefixes. A shared 30-second deadline includes requests and sleeps; only HTTP 503 with `ServiceUnavailable` is retryable, with SDK retries disabled for these probes. The actual compatibility writes, reads, multipart operations and listing assertions remain unchanged.

Add four fast regression tests to the existing PR smoke profile, with matching exclusion from the full profile. No new workflow or job is introduced.

## Verification

- `cargo nextest run --locked --profile ci -p rustfs --lib --test connect_registration --test-threads 4 -E 'binary(/^connect_registration$/) | test(=connect::diagnostics::trace_runtime::tests::local_runtime_rejects_non_private_state)' --no-tests fail --status-level pass --final-status-level fail` passed 23/23 twice in 5.088s and 5.097s: 22 macOS registration tests plus one unrelated control. JUnit intervals confirm capacity reservation; no retries or test-process leaks occurred. The Linux-only inventory case remains for CI.
- `cargo nextest run -p e2e_test --lib --profile ci -E 'test(upgrade_write_readiness_tests)' --no-fail-fast --no-tests fail` passed 4/4 twice in 1.068s and 1.072s, with zero retries. Regressions cover metadata readiness followed by write unavailability, recovery through every writer, immediate permanent-error failure despite an incoming SDK retry configuration, and deadlines for repeated 503s and stalled requests. The original failure is recorded in [the mixed-version upgrade job](https://github.com/rustfs/rustfs/actions/runs/36569700716/job/109415806473).
- `cargo fmt --all --check`, `git diff --check`, `python3 scripts/check_test_wiring.py` and compiled smoke/full membership checks passed. Smoke membership changes from 188 to 192 by adding exactly these four tests; full membership is unchanged. The expected Linux smoke digest was derived from the actual prior Linux listing plus those four platform-independent additions and still requires confirmation by this PR's CI.

Local verification covers the exact source committed in `9b2ea836317ed035a91d1fc9fcf725f70c3098e2` on main `380e98a42cb4fcd0994fed79b30c2c7605deb0bc`. An independent final-diff correctness and reliability review found no findings. Fresh Linux workspace and real mixed-version upgrade runs are required before treating the remediation as fully verified; local fake-target tests do not establish that result.

## Impact

Test scheduling and readiness only; no production behavior, API, dependency, test deadline or compatibility assertion changes. Reserving capacity serializes registration fixture processes within a nextest run. The bounded readiness probes may add startup time while peer write health converges; permanent errors still fail immediately.

## Additional Notes

Rollback by reverting this PR. Existing CI restructuring from #8233 is independent of these follow-up fixes.
2026-09-30 10:06:09 +08:00
Chris 9e33d54269 fix(release): package stable preview tags without updating channels (#8256) 2026-09-30 09:49:27 +08:00
Hauser 498080dec4 fix(storage): default new bucket durability to strict (#8253)
* fix(storage): align new bucket durability with strict defaults

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): defer failure recovery until scan completion

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-30 09:23:32 +08:00
Hauser 296854c3d2 test(heal): cover degraded MRF replay after deletion (#8249)
* test(heal): cover deleted versions during MRF replay

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): exercise degraded MRF replay after delete

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): route quorum errors through test storage API

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-30 09:14:02 +08:00
Jason Kossis 56468fd553 fix(scanner): resume checkpoints across cycle and leader changes (#8252) 2026-09-30 08:18:20 +08:00
Chris 50aa482136 fix(connect): add non-Unix health runtime fallback (#8247) 2026-09-30 08:10:10 +08:00
Chris e870a6d25b fix: separate CPU profile stack lifetime from accumulator (#8250) 2026-09-30 01:36:14 +08:00
Chris 5851d9eb54 fix: filter CPU profile samples to reviewed executable symbols (#8248) 2026-09-30 01:06:24 +08:00
Chris 1e7065101d feat: Capture offline CPU profiles in serving process (#8245)
feat: capture offline CPU profiles in serving process
2026-09-29 22:37:10 +08:00
Chris 380e98a42c fix(test): synchronize top-disk fixture writes with sampling (#8243) 2026-09-29 21:07:45 +08:00
Chris a88225d208 fix(ci): reduce duplicate work and preserve reliable test failures (#8233)
* fix(ci): reduce duplicate work and preserve reliable test failures

* fix(ci): retain protocol evidence and repair stale test fixtures

* test(connect): honor parent deadline during API fixture readiness

* fix(ci): reserve IO capacity for state writer proofs

* test(connect): align RPC fixtures with service capture contracts

* test(connect): cover pinned service capture failures
2026-09-29 21:06:45 +08:00
Chris 1cca0dd25c feat(connect): select offline key for client performance (#8242) 2026-09-29 20:49:59 +08:00
Chris b0a54f0e5c feat(connect): capture offline network performance in server (#8240)
* feat(connect): capture offline network performance in server

* fix(connect): use storage facade in RPC test

* fix(connect): keep RPC test body behind storage facade
2026-09-29 19:41:43 +08:00
Chris 83a2d8b02d feat(connect): capture RPC activity in offline server runtime (#8237) 2026-09-29 18:16:54 +08:00
Chris f661651aad feat(connect): capture API activity in offline server runtime (#8234) 2026-09-29 17:18:26 +08:00
Chris 9298991d14 fix(scanner): restart a finished mixed sweep under its requested plan (#8231)
A bucket sweep that ends mixed clears its position and records the finishing cycle's plan as started. The next cycle requests a new plan whenever the bucket was written in between, so a fresh sweep inherited a stale started plan and ended mixed again. On a continuously written bucket no sweep could ever certify and the census kept the old root.

Start a fresh verification sweep under the requested plan when no durable position remains. Resumed sweeps keep their started plan, so a clean tail still cannot certify an old prefix.

Refs #7108
2026-09-29 16:48:58 +08:00
Chris 3c7d85420c fix(test): control clocks in Connect network fixtures (#8228) 2026-09-29 16:48:34 +08:00
Chris 6a7880c9fc feat(connect): capture offline lock metrics in server (#8230) 2026-09-29 16:29:07 +08:00
hector 0e58bd09d0 fix(ci): run security chain evidence from the lane's own checkout (#8229)
The security workflow checks the repository out into rustfs-repo/ (#7212)
but its chain evidence steps still invoke
scripts/functional_chain_evidence.py relative to the workspace root, which
on the persistent shared runner resolves to a stale checkout left by another
job. The evidence gate then compares that checkout's HEAD against the chain
workflow_sha and rejects the lane before any case runs
("lane checkout differs from chain workflow source").

Run the evidence script from the lane's own checkout so ROOT resolves to
rustfs-repo/, whose HEAD is exactly the chain-pinned workflow_sha.
2026-09-29 16:09:31 +08:00
Hauser fcb60e7322 fix(rpc): preserve walk_dir missing-path errors (#8226)
## Related Issues

rustfs/rustfs#8217 and rustfs/backlog#2683.

## Summary of Changes

Reviewed 539ed275db against d2175d1e1e. No findings. Capable callers requesting missing-path reporting recover explicit pre-output FileNotFound or VolumeNotFound errors; partial-output and other failures continue to fail the stream.

## Verification

Two independent source reviews completed across all lenses below; the frozen diff matches Git and passes diff whitespace checks. Traced the authenticated handler, bounded first-chunk preflight, cancellation ownership, HTTP status/token classification and quorum error handling. The new tests cover ordinary streaming, typed missing errors, byte preservation and mixed missing/I/O failures.

Author-reported focused tests and Docker reproduction were not rerun or independently audited in this review. The reported Docker comparison predates the final report_notfound-only gate; its scope is not distributed acceptance of the final head.

## Impact

Correctness: no findings; missing errors remain typed only before output. Security/trust: no findings; authentication, body digest and operation/status/token restrictions remain. Compatibility: no findings; legacy clean EOF remains, with the documented old-server limitation. Concurrency/durability: no findings; receiver ownership cancels an abandoned producer. Simplicity: no findings. Coverage: no blocking gap identified. Performance: no findings; preflight retains one bounded chunk and ordinary report_notfound=false streams start immediately.

## Additional Notes

This review does not establish that the patch fixes any current main CI failure. Full main CI and release acceptance remain separate gates.
2026-09-29 15:29:59 +08:00
Hauser 0a61f4a81e fix(heal): park healthy legacy MRF intents (#8225)
## Related Issues

rustfs/backlog#2682 and rustfs/rustfs#8192.

## Summary of Changes

Reviewed ca970f55ec against d2175d1e1e. No blocking code findings. The hold requires a completed healthy legacy check and an exact incarnation, lease, object, version and scope; it preserves the durable journal and rechecks on restart.

## Verification

Two independent source reviews completed, covering the lenses below. The frozen diff matches Git and passes diff whitespace checks. Reported Cargo and Docker results were not rerun or independently audited in this review.

One factual correction to the PR description: `legacy_sigkill_replay_repairs_without_releasing_unverified_responsibility` uses the default four-member fixture, with three replicas before restoration and four afterward (`mrf_partial_write_test.rs:559,567`). That named regression exercises the production manager path, but is not EC12+4. Other tests in the file use sixteen disks.

## Impact

Correctness: no findings; proofless legacy health never becomes a verified receipt. Security/trust: no findings; exact identity fences remain. Compatibility: no findings; journal encoding remains unchanged. Concurrency/durability: no findings; replay retains one checkpointed owner and replacement generations become retryable. Simplicity: no findings. Coverage: no blocking gap identified, with the test-scope correction above. Performance: no findings; held entries leave the retry index without adding per-admission queue scans.

## Additional Notes

The current main journal failure concerns DecodeFailure, which follows the separate ECDecode task path. This review does not establish that this PR fixes that failure or the scanner-cycle failure, and does not establish a passing main CI or release gate.
2026-09-29 15:28:46 +08:00
Chris d2175d1e1e fix(test): keep bucket disk faults across reconnects (#8224) 2026-09-29 14:07:30 +08:00
Chris 2019715d1a Fix fresh capacity probes for formatted local disks (#8222) 2026-09-29 12:23:06 +08:00
Chris 31f05a44af fix(test): use expect_err for offline state root rejection (#8220) 2026-09-29 11:22:16 +08:00
Chris a1724f3dbe feat(connect): capture signed offline service health (#8219) 2026-09-29 10:50:23 +08:00
Chris c62979a45b fix(connect): secure new offline state roots (#8218) 2026-09-29 10:13:58 +08:00
Chris e14eacc5ed fix(rpc): keep legacy format reads in their original namespace (#8216) 2026-09-29 09:24:39 +08:00
Chris 47af5565c7 fix(ci): preserve cluster startup failure evidence (#8213) 2026-09-29 08:50:09 +08:00
Chris f0ce628a7f feat(connect): capture offline native threads in service process (#8215) 2026-09-29 08:48:56 +08:00
Chris 32cc1eb76c Capture offline memory profile from the running service (#8214)
feat(connect): capture offline memory from the running service
2026-09-29 08:43:25 +08:00
Chris 84fe13989e fix(ecstore): retain read quota and control reserve test hedging (#8211) 2026-09-29 07:16:31 +08:00
Chris c8f42b8dca fix(ci): isolate scanner deadline and expose test failure details (#8210)
* fix(ci): isolate scanner deadline fixture and expose readiness errors

* test(ecstore): report unexpected capacity reservation errors
2026-09-29 07:16:17 +08:00
Chris c9acf01fd0 fix(tier): reread mutation intents after lock contention (#8209) 2026-09-29 07:16:01 +08:00
Chris 87e9a84a4e fix(ecstore): preserve RPC status when cloning storage errors (#8207)
* fix(ecstore): preserve RPC status when cloning storage errors

* test(heal): retain clone coverage without redundant ownership
2026-09-29 07:15:46 +08:00
Chris af0ae798f8 fix(test): serialize replacement_bucket_metadata heal test under parallel load (#8212) 2026-09-29 07:07:31 +08:00
dependabot[bot] a2bdcf497b build(deps): bump the dependencies group with 14 updates (#8176) 2026-09-29 01:59:07 +08:00
Chris 5564932f38 fix(heal): retry cancelled internode RPC operations (#8205) 2026-09-29 01:58:35 +08:00
Chris 8778d55e46 fix(ci): preserve diagnostics for rio storage test failures (#8204) 2026-09-29 00:15:26 +08:00
Chris 5ebc1e480b test: stabilize heal timing and improve heartbeat replay coverage (#8150)
fix: preserve heartbeat replay and stabilize timing tests
2026-09-28 22:07:15 +08:00
Chris 4d782141d8 fix(test): add missing unavailable_drives field to readiness test initializers (#8200)
Commit 151103a609 added the  field to
StorageReadinessDetails but did not update two test initializers in
readiness.rs, causing E0063 compile errors in --all-targets builds.
2026-09-28 21:26:42 +08:00
Hauser 3bc36acf7f fix(notify): persist Docker queue stores and expose open errors (#8201)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-28 21:24:12 +08:00
Chris 4a91c75152 fix(ci): restore checks and multipart migration retries (#8196)
* fix(ci): restore multipart lint and full E2E membership checks

* fix(ecstore): preserve multipart completion identity during migration
2026-09-28 21:22:49 +08:00
Chris 2648776be7 fix: accept catalog files in health service artifact (#8198)
* fix: accept catalog files in health service artifact

* ci: retain configured health acceptance runner
2026-09-28 20:45:26 +08:00
RustFS 9442e89f5f fix(iam): require an explicit permission for force-delete (#8154)
* fix(iam): require an explicit permission for force-delete

A force-delete header no longer inherits s3:* or consoleAdmin. Bucket
force-delete requires s3:ForceDeleteBucket whenever the header is present,
and recursive object force-delete requires s3:ForceDeleteObject. A plain
delete keeps the existing checks.

Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* test(e2e): keep force-delete header names static

The bucket force-delete helper must pass a static header name into the
SDK request mutator.

Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* test(e2e): move the force-delete header into the request mutator

The SDK request customizer requires a static header name owned by the
closure.

Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* fix(iam): keep force-delete out of NotAction grants

NotAction now uses plain wildcard matching, so NotAction "s3:*" still
excludes force-delete. An Allow statement grants s3:ForceDeleteObject or
s3:ForceDeleteBucket only when its Action list names the action; a
NotAction-only Allow never does. The rule applies to both IAM and bucket
policy statements.

Also build the invalid-header errors with S3Error::with_message to keep
the s3s footprint at its baseline, and fix a clippy single_match.

---------

Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-09-28 19:08:33 +08:00
RustFS fc5609bbb0 fix(s3): make retried CompleteMultipartUpload idempotent (#8153)
Record the upload id on the completed object so a lost-response retry
returns that object's ETag instead of NoSuchUpload, while a different
part list or a replaced object keeps the existing errors.

Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-28 19:08:10 +08:00
Chris 64bd224b3b fix(connect): accept the health-era heartbeat and measured Top windows (#8195)
Two Connect tests fail on main in every Test and Lint lane.

Heartbeat: #8155 replaced "today's capabilities minus
profile.memory.service@1" with "the pre-health set minus it". That drops
the release that advertised health.check.service@1 but not yet in-service
memory profiling, so a pending heartbeat persisted by that release now
fails validation and the runtime stops. Accept that release as its own
frozen list.

Top disk/net: execute_*_job sets max_duration_millis to the requested
duration, then the capture compared the measured sleep against it. The
timer only overshoots, so a job that ran exactly as requested was
rejected with LimitExceeded whenever the overshoot reached 1 ms. Report
the authorized window, as Top API already does; validate_capture bounds
it by the limit.
2026-09-28 19:07:39 +08:00
Chris 71f8b607ad fix(ci): restore mainline and scheduled test reliability (#8187)
* fix(ci): restore mainline and scheduled test reliability

* fix(ci): provide GitHub CLI for CPU acceptance

* ci: provide Docker for CPU service acceptance

* ci: provision Python and Docker for OIDC validation

* ci: restore hosted runners for Docker validation

* ci: use verified MinIO release packages for interop

* ci: preserve host ownership of MinIO fixtures

* fix(ci): correct diagnostic limits and isolate startup checks

* test(connect): include object CLI failure details

* test(readiness): initialize unavailable drive diagnostics
2026-09-28 19:07:29 +08:00
GatewayJ cef0c61532 fix(table-catalog): preserve data sequences during file rewrites
Merge the reviewed fix from pull request #8104.
2026-09-28 18:51:58 +08:00
Justin Bradfield fc325bcc56 fix(s3): report x-amz-mp-parts-count on GetObject with partNumber
Merge the reviewed fix from pull request #8115.
2026-09-28 18:51:48 +08:00
cxymds 4a46f5ff3d fix(s3): serve bucket websites on dedicated domains
Merge the reviewed fix from pull request #8183.
2026-09-28 18:48:21 +08:00
cxymds e0a973bf4e fix(s3): preserve prefix listing pagination and marker metadata
Merge the reviewed fix from pull request #8181.
2026-09-28 18:47:03 +08:00
hector 671238458c ci: fail the pool lane when expected step markers are missing
Merge the reviewed fix from pull request #8190.
2026-09-28 18:46:12 +08:00
Chris 151103a609 fix(ecstore): recover remote disks after transient stalls
Merge the approved fix from pull request #8149.
2026-09-28 18:40:30 +08:00
Nikita Bakun cc7d5e3f5f fix(replication): stop sending versionId query once a target rejects it
Merge the approved fix from pull request #8109.
2026-09-28 18:40:20 +08:00
yi111 d556d4dc94 fix(admin): report a missing policy or user as 404 NoSuchResource, not 500 InternalError
Merge the approved fix from pull request #8127.
2026-09-28 18:40:10 +08:00
chapman 6300fbe74f fix: map IAM not-found errors to 404 in InfoCannedPolicy and RemoveUser
Merge the approved fix from pull request #8134.
2026-09-28 18:38:20 +08:00
Chris 6ebc78d115 fix(ci): run profile acceptance on Docker runners (#8191) 2026-09-28 16:14:07 +08:00
Chris 5f4d5e8fe8 fix(ci): run CPU acceptance with Docker
Use the Docker-enabled runner required by the connected CPU service-job test.
2026-09-28 15:38:36 +08:00
Chris e4520f70a1 fix(ci): verify CPU acceptance identity without gh
Use authenticated curl for the pinned release run and artifact checks on sm-standard-2.
2026-09-28 15:13:03 +08:00
hector f54945a4c4 fix(ci): repair pool budgets and Connect test failures (#8155)
* ci: give the pool run a real wall-clock budget

The pool suite budgets up to 24h each for rebalance and decommission
(REBALANCE_TIMEOUT / DECOMMISSION_TIMEOUT in rustfs_pool_expand.sh),
but the workflow step capped it at 45 minutes. Four consecutive runs
died identically: steps 1-7 all PASS, then the rebalance wait was
killed at exactly 45:12 - chain 35680308150, standalone 35702803577,
chain 35757773373, chain 36283120750 - with rebalance at completed=1/4
(~21 minutes in), so a full pass has never been observed.

Make the budget an input (default 240 minutes: covers the observed
rebalance pace plus one decommission pass) and document why. The suite
keeps failing the job through its [POOL-STEP] marker adjudication.

* fix(ci): align pool job budget and timeout contract tests

* fix(connect): preserve legacy heartbeats and isolate I/O tests

---------

Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: RustFS <hello@rustfs.com>
2026-09-28 14:26:21 +08:00
Hauser 4beac12439 fix(connect): return profile capture result directly (#8182)
* fix(connect): return profile capture result directly

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* style(connect): normalize profile capture formatting

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(tests): use async mutex for thread profiles

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(connect): widen diagnostic timing budgets

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-28 14:25:27 +08:00
Chris 6523ab8d9f fix(ci): verify GNU packages with CPU catalogs
Match the archive-member check to the GNU pyroscope catalog packaging condition.
2026-09-28 13:23:46 +08:00
Chris 0609e7ce14 fix(connect): admit complete release CPU symbol catalogs
Bound the release catalog to 32 MiB and 200,000 symbols based on the GNU build measurement; keep complete names and reject catalogs beyond either limit.
2026-09-28 12:09:26 +08:00
Chris afb9ddabb4 fix(helm): route Gateway API traffic to the chart service (#8145)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-28 02:26:56 +00:00
RustFS 2e014d25b3 fix: bound restarted-peer stalls on quorum reads and writes (#8152)
A restarted peer can accept a pooled connection and never send response
headers, so HttpReader::open waited past the client body timeout before
the body-stall timer or erasure hedge could run. Bound that header wait
by the stall timeout and retry the open once on a fresh connection.

After write quorum, MultiWriter still waited out the full disk stall
for a silent peer, which matches the client timeout. Give remaining
writers one second, then drop them so the caller returns.

Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2026-09-28 02:22:41 +00:00
Chris 719000460b ci(connect): match GNU build runner in acceptance checks (#8180) 2026-09-28 09:08:25 +08:00
hector 528a368144 ci: refresh the preview-release contract for the current build workflow (#8178)
The contract check asserts exact lines in build.yml, and five recent
build-workflow PRs drifted it out of sync, so Security Audit's Workflow
Pin Report job fails for every PR and every push to main since Sep 27
17:34Z (first failure on push run 36337450094). Refresh the assertions
to the current, intentional shapes - contract intent is preserved and
in fact tightened:

- #8160 added PROFILE_ARGS to the cross/native cargo build lines.
- #8171 added 'should_publish == true' as a stricter prefix on the R2
  publication guard and the create-release / upload-release-assets /
  publish-release / update-latest-version conditions.

Verified locally: the full check (including the docker and helm
workflow contract tests) passes against main's files with these
assertions.
2026-09-28 08:54:33 +08:00
Chris d791d7b761 ci(connect): use Node 25 for acceptance workflows (#8179) 2026-09-28 08:52:32 +08:00
Chris 1a251de4f3 feat(connect): execute signed disk diagnostics in the service (#8177) 2026-09-28 08:36:49 +08:00
Chris e634df8611 fix: collect offline Top disk from the running service (#8174)
fix(connect): capture offline disk IO in the running service
2026-09-28 01:35:40 +08:00
Chris f5fbb5f4e1 ci: verify Top disk in the serving process (#8173) 2026-09-28 01:34:29 +08:00
Chris 5d2d539423 ci: default manual builds to artifacts without publishing (#8171)
Default manual builds to artifacts without publishing
2026-09-28 01:10:52 +08:00
Chris acebc9649d feat(connect): capture service runtime profiles over local IPC (#8168)
feat(connect): export service runtime profiles over local IPC
2026-09-28 00:49:45 +08:00
Chris e33542b0c2 fix(scanner): recover after cycle-state persistence failures (#8148)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 16:39:41 +00:00
Chris 2ee0ad1704 Collect bounded metrics from the serving Tokio runtime (#8167)
feat(connect): collect bounded service runtime profiles
2026-09-28 00:20:19 +08:00
Chris 4c5fc3c991 Use pinned release catalogs for Connect CPU acceptance (#8166) 2026-09-28 00:18:41 +08:00
Chris 6d2b1c629e Generate CPU symbol catalogs from release executables (#8165) 2026-09-28 00:12:36 +08:00
RustFS de484b93e0 fix: admit cold reads and readiness at read quorum (#8156)
EC 2+2 still meets read quorum with two of four nodes up. A survivor
that had not cached a bucket returned 503 on GET, HEAD, and List, and
/health/ready left the Service once write quorum was lost. Reads and
Service membership now follow read quorum and shared locks. Writes and
/minio/health/cluster still require write quorum and exclusive locks.

Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 13:58:55 +00:00
hector 4f61b007da ci: build linux-aarch64 natively on sm-standard-4-arm and verify every Linux package (#8160)
Route the two linux-aarch64-* release legs to the dedicated arm64 runner
and build them natively (cross: false) instead of cross-compiling via
cargo-zigbuild on x86_64 hosts.

With native legs now executable, extend the packaged-artifact checks that
were previously gated to x86_64-unknown-linux-gnu only:

- Every Linux leg runs rustfs --version and rustfs-cli --help from its own
  package, proving the artifact matches the runner architecture.
- The packaged-console runtime smoke (server boot + console HTTP 200) also
  covers aarch64-unknown-linux-gnu, so both architectures get full runtime
  verification. Musl legs run the binary checks but skip server boot for
  now.
2026-09-27 13:40:23 +00:00
cxymds 1dd354233c test(connect): sync heartbeat capability expectations (#8161) 2026-09-27 12:22:48 +00:00
Chris 6eb14a11d0 fix(ci): run health acceptance on Docker-capable workers (#8158) 2026-09-27 14:26:28 +08:00
Chris 3667eea893 fix(ci): validate health binary without file utility (#8157) 2026-09-27 14:22:49 +08:00
hector e67bbd7bd3 ci: run the chain orchestration jobs on the smoke-testing runner (#8147)
* ci: run the chain orchestration jobs where the gh CLI exists

#8112 moved these jobs to the sm-standard-2 pool, whose images ship
without the gh CLI (a known property of the build fleet, see #7572).
Last night's chain died in prepare before any suite ran:
resolve_functional_candidate.py:34 does subprocess.run(["gh", "api"])
and got FileNotFoundError; complete-chain and the hourly
functional-chain-health canary fail the same way. On Sep 22/23 the same
jobs ran green on GitHub-hosted runners.

Move prepare, complete-chain and the health canary to ubuntu-latest.
The suite lanes stay on smoke-testing.

* ci: run the chain orchestration jobs where the gh CLI exists

#8112 moved these jobs to the sm-standard-2 pool, whose images ship
without the gh CLI (a known property of the build fleet, see #7572).
Last night's chain died in prepare before any suite ran:
resolve_functional_candidate.py:34 does subprocess.run(["gh", "api"])
and got FileNotFoundError; complete-chain and the hourly
functional-chain-health canary fail the same way. On Sep 22/23 the same
jobs ran green on GitHub-hosted runners.

Move prepare, complete-chain and the health canary to ubuntu-latest.
The suite lanes stay on smoke-testing.

* ci: point the chain orchestration jobs at the smoke-testing runner

Per maintainer decision, consolidate them onto the same runner as the
test lanes. Verified on rustfs-smoke-testing as the runner user:
gh 2.45.0, jq 1.7.

* ci: point the chain orchestration jobs at the smoke-testing runner

Per maintainer decision, consolidate them onto the same runner as the
test lanes. Verified on rustfs-smoke-testing as the runner user:
gh 2.45.0, jq 1.7.
2026-09-27 08:26:43 +08:00
Chris 8d033c85d8 fix(ci): install GitHub CLI for Connect health acceptance (#8146) 2026-09-27 08:04:54 +08:00
Chris b7420fe5f2 fix(ci): bootstrap release upload CLIs (#8144)
fix(ci): bootstrap release upload CLIs on self-hosted runners
2026-09-27 08:00:48 +08:00
Chris 5cb1c9e8cb fix(architecture): route health.rs storage imports through storage_api facade (#8143)
The health diagnostics module imported directly from
crate::storage::storage_api, bypassing the architecture migration
guardrail. Add a connect facade module to rustfs/src/storage_api.rs
and redirect the imports. Also fix a clippy::redundant-guards lint
in the coarse-flags match arm.
2026-09-27 07:53:54 +08:00
Hiroaki KAWAI 20356ab709 fix(scanner): build raw enumeration indexes incrementally (#8114)
* fix(scanner): build raw enumeration indexes incrementally

* test(scanner): iterate over restart test entries directly

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 07:49:59 +08:00
Chris 2c5c43b0e0 test(connect): expose typed drive measurement failure context (#8099)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 05:11:47 +08:00
Chris cf096b3c22 fix(connect): pace diagnostic network payload sends (#8096)
* fix(connect): pace diagnostic network payload sends

* test(connect): exercise native network pacing entry

---------

Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 05:11:35 +08:00
Chris 82024907c5 test: add health service acceptance workflow (#8140) 2026-09-27 03:55:17 +08:00
Chris e2785846d0 ci: keep gh-dependent release jobs on GitHub-hosted runners (#8137)
The sm-standard-2 fleet has no gh CLI (see #7572), so moving the release publication jobs onto it in #8112 broke every tag release: the 1.0.1-preview.12 build failed in Create GitHub Release with 'gh: command not found'. Move create-release, upload-release-assets, publish-release, cleanup-preview-releases, and package.yml resolve/package back to ubuntu-latest.
2026-09-27 03:37:23 +08:00
Chris 31ac243865 feat(connect): add bounded health service job 2026-09-27 02:50:49 +08:00
Chris 8741bb77ea fix(storage): weight automatic multipart admission by part size (#8118)
* fix(storage): weight automatic multipart admission by part size

* fix(ci): restore filesystem runner capabilities and typos dependency

* fix(ci): make release guard portable and spell out part variables

* ci: restore sm-standard-2 runners for io_uring and distributed e2e jobs

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-26 22:35:39 +08:00
cxymds 212b20f890 fix(ecstore): admit cold bucket metadata at read quorum (#8120)
* fix(ecstore): admit cold bucket metadata at read quorum

Encode a durable creation-commit state in bucket metadata so committed Object Lock buckets can use read quorum for cold loads while pre-physical creation intents remain fail-closed. Drain commit-write fan-out, keep write quorum for uncommitted intents, and add regression coverage for exact, below, migrated, and partial-create quorum boundaries.

* fix(ecstore): fence bucket creation-commit persistence

Address review findings on the creation-commit proof.

Persist the proof only under the bucket metadata transaction fence: at bucket creation, and through a fenced migration that re-reads the authoritative metadata and revalidates physical presence at write quorum before writing. This stops a stale snapshot from reverting an acknowledged configuration update or outliving a delete/recreate.

Establish commitment when Object Lock is enabled on an existing bucket, inside the same configuration mutation, so a healthy cluster no longer rejects object operations with ErasureWriteQuorum.

* fix(ecstore): keep commit fence error message stable

The error(format!) ratchet requires a stable Display for quorum bucketing; use a fixed message instead of embedding the bucket name.

* test(ecstore): reuse canonical Object Lock fixture in regression tests

The s3s footprint ratchet is shrink-only. Use the existing ENABLED_OBJECT_LOCK_CONFIG static instead of naming s3s DTO types in store tests.

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-26 21:40:40 +08:00
hector 7d3d0b6c2b ci: align the matrix contract with the 36-combination sweep (#8136)
Companion to rustfs/auto-testing#109: the 4x1 topology is removed from
MATRIX_TOPOS because the product rejects a 1-drive-per-node distributed
pool at startup (FATAL BelowMinimum). The unfiltered sweep is now 36
combinations / 300 cases; update the contract totals and comments so an
unfiltered sweep is still held to an exact expected case count.
2026-09-26 20:31:32 +08:00
hector d317bb3274 ci: run the KMS Vault lanes on ubuntu-latest again (#8133)
#8112 replaced ubuntu-latest with sm-standard-2 across workflows. Both nightly
KMS Vault lanes need a working Docker daemon; the sm-standard-2 pool does not
provide one, so both lanes die in seconds on 'Docker is not available' while
the package compiles and publishes fine - and the functional chain never fires
because its gate requires a successful build run. Restore the lanes to
ubuntu-latest, as documented in the lane comment before #8112.
2026-09-26 19:12:18 +08:00
Hauser 5cac685e9a feat(ecstore): integrate io_uring backend improvements (#8105) 2026-09-26 17:14:17 +08:00
Hauser d8bf268885 fix(scanner): bound SNSD deep scans and checkpoint cloning (#8126)
* fix(scanner): bound SNSD deep scans and checkpoint cloning

* ci: run mount-dependent jobs on hosted VMs

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
2026-09-26 06:06:50 +00:00
hector fdd753a1dd ci: run distributed e2e on ubuntu-latest to restore tmpfs mounts (#8128) 2026-09-26 12:00:26 +08:00
Chris de80e8a673 fix(ci): make shared checks portable across runners (#8124)
* fix(ci): isolate monitor argument checks from runner tools

* fix(ci): install the Typos action download dependency

* fix(ci): make release policy matching portable across awk variants
2026-09-26 10:55:06 +08:00
RustFS 4cf45e9ed2 Update bug report storage requirements: allow SAN/JBOD, recommend XFS (#8125)
docs(github): clarify supported storage in bug reports

State that local disks, SAN/Fibre Channel volumes, and JBOD are supported,
recommend XFS, and keep the existing unsupported remote/shared filesystem rule.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-09-26 10:43:33 +08:00
Chris 9f8acf688c feat(github): add issue forms with storage support checks (#8123) 2026-09-26 10:07:49 +08:00
Chris 96fc26cd6a fix(release): defer installation updates until artifacts are live (#8122) 2026-09-26 09:17:15 +08:00
hector edcc81a8fd ci: replace ubuntu-latest runner with sm-standard-2 across workflows (#8112)
* ci: replace ubuntu-latest runner with sm-standard-2 across workflows

* ci: keep scheduled-validation monitors on hosted runners

The freshness and watchdog jobs report stalled scheduled validations.
Running them on the same sm-standard-2 pool means a pool outage stalls
the monitors too, so nothing reports it.

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-09-25 10:02:45 +08:00
Hauser 286bbb8e1f chore: upgrade workspace dependency releases (#8107)
* chore: upgrade workspace dependencies

* chore: refresh dependency lockfile

* fix(rebalance): avoid waiting on active entry gate during stop preparation
2026-09-24 21:24:14 +08:00
cxymds 0913686b12 fix(heal): detect stale delete-marker metadata (#8102) 2026-09-24 11:16:02 +08:00
George Melikov df64242f64 fix(ecstore): resume scan_dir past a dash-suffixed sibling directory (#8071)
A flat (recursive) listing resumed inside "s/" re-emitted everything in
its sibling "s-x/". scan_dir dropped the entries before forward_to by
comparing directory names without their trailing slash, where "s" sorts
before "s-x", while the keys they stand for sort the other way round:
'-' (0x2d) is below '/' (0x2f), so all of "s-x/..." precedes "s/...".
The drain stopped at "s" and kept "s-x".

The next page then started with keys at or before the marker; the
listing layer filtered all of them out, found no more candidates and
answered IsTruncated=false. A bucket of 104,137 objects with backup
directories named "<id>" and "<id>-rollbacks" listed as 5,000; two such
directories of 1,200 keys each listed as 2,000.

Compare every remaining entry as the key prefix it stands for, slash
included, and keep it only when it sorts at or after forward_to or
contains it. The remainder of forward_to is taken before `current` is
trimmed, so this also holds below the bucket root; retain also drops
entries when all of them precede forward_to, which the drain never did.
2026-09-24 07:13:30 +08:00
Chris d3b75e695e test(connect): add scheduler receipt acceptance workflow (#8078)
* test(connect): add scheduler receipt acceptance workflow

* chore(deps): upgrade crates and fix faster-hex advisory (#8098)

* test(connect): isolate scheduler acceptance workflow

---------

Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-24 03:34:27 +08:00
Chris dde1aa77a1 fix(connect): preserve diagnostic schedules across restart (#8092) 2026-09-24 03:28:37 +08:00
Chris be01e513be feat(connect): sample memory within the running service (#8091)
* feat(connect): sample memory within the running service

* test(connect): add official memory service acceptance

* test(connect): consume the final memory job without cloning
2026-09-24 03:25:28 +08:00
cxymds 9ab034df62 fix(heal): preserve retryable writer failures (#8089)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-24 02:41:16 +08:00
Tyler Hillery 624463c5ba fix(s3): default CopyObject content-type to binary/octet-stream on REPLACE (#8073)
fix: default CopyObject content-type to binary/octet-stream on REPLACE
2026-09-23 18:04:05 +00:00
Chris 713920dcd6 docs: update license copyright to RustFS, Inc. (#8097) 2026-09-24 00:56:08 +08:00
GatewayJ f219a0aba8 perf(ecstore): parallelize multipart I/O setup and metadata reads (#8085)
* perf(ecstore): parallelize multipart I/O setup and metadata reads

* perf(ecstore): share multipart paths and increase read concurrency

* fix(ecstore): route multipart benchmark through storage API facade
2026-09-23 22:19:23 +08:00
Hauser 311e4306e6 chore(ci): complete action pin and HAProxy upgrades (#8090) 2026-09-23 19:20:39 +08:00
Hauser 7f0f4941cf chore(ci): refresh checkout pin in validated workflows (#8083)
* chore(ci): update checkout pin in validated workflows

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): pin migrated checkout alerts to requested commit

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(ci): pin additional checkout workflows to verified commit (#8084)

* chore(ci): pin more checkout workflows to requested commit

Update six additional workflows to the verified upstream checkout commit without changing their permissions, inputs, or triggers.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(ci): pin functional and OIDC checkout uses

Extend the verified checkout commit pin to functional-chain and OIDC workflows without changing their behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-23 09:27:54 +00:00
cxymds 655bb470b8 fix(heal): retry faulty storage disk errors (#8086) 2026-09-23 07:48:39 +00:00
cxymds 56a4099f99 fix(heal): recreate missing bucket volumes (#8082)
* fix(heal): recreate missing bucket volumes

* fix(heal): use typed error for empty targets
2026-09-23 14:51:24 +08:00
hector ee6de7d786 ci(package): build gnu and musl DEB/RPM variants with distinct file names (#8079)
* ci(package): build gnu and musl DEB/RPM variants with distinct file names

The Build and Release workflow produces four Linux binaries
(x86_64-gnu, aarch64-gnu, x86_64-musl, aarch64-musl), but packaging
only consumed the two gnu artifacts. Add matrix entries for the two
musl artifacts so every release ships all four DEB/RPM variants.

The libc variant is now part of the package file names, which would
otherwise collide between gnu and musl builds of the same version:

- deb: rustfs_<version>_<libc>_<arch>.deb
- rpm: rustfs-<libc>-<version>-<release>.<arch>.rpm

The dpkg Package and rpm Name stay plain "rustfs", so gnu and musl
remain mutually exclusive upgrades of one package rather than
co-installable packages fighting over /usr/bin/rustfs.

Dependency declarations now follow the linkage: gnu binaries
dynamically link glibc and keep Depends: libc6 (>= 2.31) /
glibc >= 2.31; musl binaries are statically linked and declare no
libc dependency. The libc variant is also visible in the package
description.

scripts/release/package_versions.sh gains a LIBC argument and its
contract tests cover both variants plus the invalid-libc cases.

* ci(package): align deb/rpm file names with the zip artifact naming

Rename the package file names so every release asset of one build
shares the same stem as its binary artifact, differing only by
extension:

- before: rustfs_<deb_version>_<libc>_<deb_arch>.deb
          rustfs-<libc>-<rpm_version>-<rpm_release>.<rpm_arch>.rpm
- after:  rustfs-linux-<arch>-<libc>-v<version>.deb / .rpm

e.g. rustfs-linux-x86_64-gnu-v1.0.0.zip,
     rustfs-linux-x86_64-gnu-v1.0.0.deb,
     rustfs-linux-x86_64-gnu-v1.0.0.rpm.

Non-development builds embed the raw release tag (with 'v'), like the
zips; development builds embed dev-<full sha>. The dpkg/rpm versions
(including the '~' prerelease ordering) are unchanged - they live in
the package metadata, and a side effect is that release asset names no
longer contain '~' (which GitHub normalizes to '.').

package_versions.sh now takes the target arch (x86_64|aarch64) instead
of the deb/rpm arch pair; the deb Architecture (amd64/arm64) in the
control metadata still comes from the workflow matrix. The two test
workflows that assemble deb download URLs from a release tag
(rustfs-table-test, rustfs-upgrade-test) are updated to the new name,
which also removes their '~'-to-'.' asset name workaround.
2026-09-23 03:17:47 +00:00
cxymds 1880b42169 fix(heal): preserve decode repair signals and retry startup quorum (#8067)
* fix(startup): retry transient bucket metadata quorum failures

* fix(ecstore): retain recovered shard damage for read repair

* fix(ecstore): reconnect missing read sources before GET

* test(heal): prove exact marker replay preserves history

* fix(ecstore): fence reconnect retries at the deadline

Reject expired reconnect waiters before dispatch and start cooldown on sweep completion, including timeout and cancellation. Cover exact-deadline admission independently of cooldown and preserve the original strict RPC-count regression.
2026-09-22 14:58:31 +00:00
Chris 3f549e26f3 fix(ci): publish version-tagged preview Docker images (#8068) 2026-09-22 13:06:23 +00:00
Dae-Cheol Noh 3c6c88b2e7 fix(amqp): emit S3 events directly in notification records (#8066)
* fix(amqp): emit S3 events directly in notification records

* test(amqp): avoid queued timestamp equality race
2026-09-22 12:16:40 +00:00
Dae-Cheol Noh 1f04a12abf fix(ftps): bound upload memory with multipart streaming (#8064)
* fix(ftps): bound upload memory with multipart streaming

* fix(ftps): keep multipart upload within driver module
2026-09-22 12:15:20 +00:00
cxymds 122abfaae2 feat(integrity): add inventory, audit, and protected migration (#8065)
* feat(integrity): add inventory, audit, and protected migration

* refactor(admin): use gateway facade for integrity handlers

* fix(integrity): sort fingerprint metadata explicitly
2026-09-22 19:23:41 +08:00
cxymds d0ce2f758b fix(ecstore): bound decommission target gate contention (#8061)
* fix(ecstore): bound decommission target gate contention

Target capacity gate contention during pool decommission escalated a
per-object transient into a durable bucket pause: each contended object
failed its bucket entry, which re-ran the whole bucket listing and amplified
attempts on the same objects.

- centralize the decommission capacity failure classification so gate
  contention, benign contention and fatal failures are decided once
- retry target-gate contention inline (bounded, jittered) before a mutation
  is admitted, covering put, part, complete, new-multipart and abort
- defer contended entries to the end of the round instead of failing the
  bucket entry, and require the deferred set to drain before a set completes
- treat a missing object or version, an overwrite and a superseded upload id
  as benign contention that is neither counted as a failure nor escalated
- use full-jitter exponential backoff, capped, for decommission retries
- expose the capacity pause reason, the waiting reason and a cumulative
  pause count in the admin pool status, plus gate-retry and per-object
  attempt metrics

Related: rustfs/backlog#2644

* fix(ecstore): correct deferred replay and metadata compatibility
2026-09-22 19:20:05 +08:00
hector 9c30cc8851 feat(ci): on-demand fault-tolerance matrix sweep workflow (#8058)
Adds rustfs-fault-tolerance-matrix.yml, a manually dispatched workflow that
runs the --matrix topology x EC outage sweep from rustfs/auto-testing (38
combinations, 320 cases). The scenario suite hardcodes --all and a 60-minute
budget, so the exhaustive sweep had no CI entry point.
2026-09-22 13:22:18 +08:00
Hauser 0ff06757c8 fix(obs): support custom OTLP CA bundles (#8060)
Add RUSTFS_OBS_TLS_CA_FILE for additive private CA trust in OTLP HTTP exporters.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-22 13:21:44 +08:00
cxymds e12100b4a1 fix(s3): emit x-amz-expiration as HTTP-date (#8057)
The expiry-date field of the x-amz-expiration response header was
rendered as RFC 3339 (e.g. 2026-10-02T00:00:00Z). AWS SDKs parse this
value with an RFC 822 parser; the Java SDK v1 rejects the ISO-8601
shape, drops the whole header, and logs a WARN from
ObjectExpirationHeaderHandler. S3 specifies HTTP-date (RFC 1123 with a
literal GMT zone), which is also what MinIO emits.

Render via the shared HTTP-date formatter (extracted alongside
format_expires_header) and pin the exact output in the unit test.
2026-09-22 11:18:59 +08:00
hector d7c42d4099 fix(ci): accept the testing-sha staleness fallback in chain evidence (#8056)
The 09-22 nightly chain failed all 12 lanes in seconds at the 'Bind
functional candidate' step:

  ValueError: private script pin differs from chain

resolve_functional_candidate.py's >24h staleness fallback (added by
#8026, made functional by #8041's token fix) legitimately sets
manifest.testing_sha to auto-testing main HEAD, but current_chain()
still required it to equal .config/functional-script-revision.txt -
a check written for the pre-fallback world where the two could never
diverge. Once the fallback finally fired, prepare produced testing_sha
21edcf4 while the pin file still holds 27e9584 and every lane aborted
before checking out the test scripts.

Drop the pin-file comparison and keep what the lane actually needs to
guarantee: testing_sha is a valid commit sha (current_chain), the lane
checked out exactly that sha (record: private_head == testing_sha, kept
as-is), and the health checker validates the same format instead of
re-reading the pin file. Tests updated: a fallback testing_sha that
differs from the pin is accepted; a non-sha testing_sha is rejected.

Verified: python3 -m unittest test_functional_chain
test_functional_chain_health -> 39 tests OK.
2026-09-22 10:25:36 +08:00
hector 4b0118520e test(ci): align FT workflow contract tests with scenario E (44 cases) (#8054)
#8052 renamed the fault-tolerance step to 'Run fault-tolerance
scenarios (A, B, C, C2, D, E)' and raised the evidence gate to 44
cases, but scripts/test_security_workflow.py still expected the old
step name and a 38-case fixture, breaking Quick Checks with three
KeyErrors and one assertion failure. Update DIRECT_TESTS and the
write_result fixture to the new contract.
2026-09-22 00:18:40 +08:00
cxymds 1afaf4491e fix(ecstore): bound decommission capacity gate waits (#8044)
* fix(ecstore): bound decommission capacity gate waits

* fix(ecstore): keep rebalance meta save lock errors retryable

The rebalance metadata retry policy only retries when the source chain still
carries the typed lock error, but resolve_rebalance_meta_save_result collapsed
every failure into a plain string. A transient rebalance.bin write-lock timeout
therefore bypassed retry_rebalance_metadata_access entirely and surfaced as a
hard failure of the periodic and terminal rebalance metadata saves.

Wrap the failure with the data-movement stage context instead, which keeps the
original lock error reachable through rebalance_error_source so the existing
transient lock policy applies. The rendered message is unchanged.

Add a regression test that feeds the wrapped meta save lock timeout through the
retry helper and asserts the retry engages.

* chore(ecstore): refresh error format ratchet baseline

Removing the string-wrapped meta save failure drops one `::other(format!`
call site in the rebalance worker, so the shrink-only baseline must be
regenerated in the same change.
2026-09-21 20:56:26 +08:00
hector bdea44f332 fix(ci): align FT evidence contract with scenario E (44 cases) (#8052)
The fault-tolerance suite now registers and runs scenario E
(rustfs/auto-testing#100): --all covers A, B, C, C2, D, E with 38 + 6
= 44 expected cases. The chain-evidence assertion still hardcoded
[A, B, C, C2, D] and 38 everywhere, which would fail every green FT
run's evidence validation once E is registered.

Bump the scenario list to include E and all count assertions from 38
to 44; step name updated to match. No other lanes reference 38.
2026-09-21 20:13:56 +08:00
cxymds 8ba3309624 fix(rebalance): retry fleet proof before start activation (#8049)
Retry the transient missing fleet capability proof in both the admin start flow and node RPC activation. Return a retryable 503 with Retry-After after rollback when readiness does not converge.
2026-09-21 11:27:58 +00:00
cxymds cd2fa50e50 fix: rename admin metrics endpoint to realtime (#8046) 2026-09-21 19:23:58 +08:00
cxymds 077b961232 fix(obs): default span events to none and gate span list by level (#8047)
JSON log layers synthesized span lifecycle events for every span and
embedded the full ancestor span list on every record. With INFO spans on
object hot paths that produced multi-megabyte lines, which journald
truncated at 48 KiB: the payload was silently dropped and the volume
itself became the dominant load.

Add RUSTFS_OBS_LOG_SPAN_EVENTS (none|close|full) with a default of none,
and keep with_span_list enabled only for verbose (debug/trace) levels so
the ancestor chain stops being re-serialized per record. The request_id
promotion path reads the span scope, not the span list, so it is
unaffected.

Refs rustfs/backlog#2642
2026-09-21 19:15:39 +08:00
cxymds e1bc731331 fix(lifecycle): reject empty expiration actions (#8045)
* fix(lifecycle): reject empty expiration actions

* style: apply rustfmt to lifecycle validation
2026-09-21 15:43:07 +08:00
cxymds bb43ef861a fix(decommission): use owned data for source capacity (#8038) 2026-09-21 15:29:30 +08:00
Chris 4047ed6c8d fix(heal): restore bucket metadata before replacement completion (#8042) 2026-09-21 12:18:06 +08:00
hector ae6bbaef27 fix(ci): give the chain staleness probe a token that can read auto-testing (#8041)
resolve_functional_candidate.py probes rustfs/auto-testing (private) to
age the pinned functional-script revision and fall back to main HEAD
after 24h. The prepare step passed github.token, which cannot see the
private repo, so every nightly chain logged

  staleness probe failed (...exit status 1.); keeping pinned revision

and replayed the 09-14 harness. On the 09-20 nightly that harness died
on the dpkg conffile prompt in all 12 lanes (see rustfs/auto-testing#97)
because the --force-confold and other fixes never reached the chain.

Use PF_TESTING_GH_TOKEN - already required by the other steps in this
workflow - so the probe can actually run and the >24h fallback works.
2026-09-21 11:08:41 +08:00
cxymds a7c875b48f fix(heal): preserve remote corruption and reject incomplete deep scans (#8032)
* fix(heal): reject truncated encoded shards

* test(heal): cover truncated shards across integrity modes (#8035)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): preserve remote corruption and reject incomplete deep scans

* test(admin): tolerate config lock timeout while polling

* fix(test): use admin storage error facade

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-09-21 02:33:03 +00:00
マルコメ 42dd76d3b2 ci(docker): sync the Docker Hub overview from README.md on release (#8034)
Docker Hub's overview is a separate `full_description` field that
`docker push` never touches, so it had drifted into an 8 KB snapshot of
an old README.md that still linked to
https://docs.rustfs.com/introduction.html (now 404).

Add a `sync-dockerhub-description` job to docker.yml that runs after the
images are pushed and republishes README.md from the same commit via
peter-evans/dockerhub-description (pinned to v5.0.0). It reuses the
existing DOCKERHUB_USERNAME / DOCKERHUB_TOKEN credentials, so no new
secrets are needed. Relative links (docs/, CONTRIBUTING.md) are
rewritten to github.com URLs so they resolve on Docker Hub.

Docker Hub caps the field at 25,000 bytes and the action truncates to
fit with only a warning; README.md is at 22,879 bytes today. Read the
published overview back after the sync and fail the job if it hit the
cap, so a truncated overview cannot be published silently.

Fixes #7995

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-21 07:04:47 +08:00
Chris 9aeecad4a9 docs(security): add encrypted ETag advisory lesson (#8040) 2026-09-20 23:52:47 +08:00
Hauser 3738b32df1 chore(deps): update flake.lock (#8033)
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/aff8a0b' (2026-09-10)
  → 'github:NixOS/nixpkgs/a32edd7' (2026-09-17)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/228ecef' (2026-09-12)
  → 'github:oxalica/rust-overlay/26a71e6' (2026-09-19)
2026-09-20 14:54:45 +08:00
Hauser d997b07505 fix(get): skip futile resume for single-disk objects (#8031)
* chore: pin s3s to upstream git revision

Use the upstream s3s git repository at 50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6 for both s3s and s3s-sigv4.

Allow the official s3s git source in cargo-deny so source checks continue to pass.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(get): skip futile resume for single-disk objects

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-20 13:25:08 +08:00
Chris 0632a64f91 fix(ecstore): allow server-side copy at normal read quorum (#8029) 2026-09-20 12:36:53 +08:00
hector cbf6b02905 ci(chain): fall back to auto-testing main HEAD when the pinned harness goes stale (#8026)
The functional chain pins the test harness to the revision recorded in
.config/functional-script-revision.txt, which is only refreshed by
release syncs - it went stale on Sep 16 and the nightly chain has been
replaying a Sep-17-era harness ever since. Last night that meant:
the performance lane died on a dpkg conffile prompt (fix merged as
#88 but never reached the chain), the FT lane died on 'unknown option:
--results-file' (harness/workflow version mismatch), and the
STS-105 signing fix (#94) sat untested.

When the pinned revision's commit is older than 24 hours, fall back to
auto-testing main HEAD so the chain always runs the current harness.
Any staleness-probe failure keeps the pinned revision (fail-safe).
The revision file stays in place as the audit trail and the release
sync continues to manage it.
2026-09-20 11:27:56 +08:00
hector ed2c3eaf77 fix(package): preserve service state across upgrades (#8024)
* fix(package): preserve service state across upgrades

* fix(package): match legacy DEB versions in tilde form

Published prerelease packages carry ~ in the dpkg control version
(package_versions.sh maps the SemVer prerelease - to ~), so the
legacy fallback list written with dots never matched 1.0.0~rc.x and
upgrades away from those DEBs still left the service stopped (#8011).

Fix the legacy glob to the tilde form and update the contract test,
which had enshrined the dot form. Verified on Ubuntu 24.04 systemd
containers: DEB upgrade 1.0.0~rc.5 -> 1.0.1 now keeps the service
running; 1.0.0 -> 1.0.1 and 1.0.1 -> 1.0.2 marker path still pass.

* docs(package): expand /etc/default/rustfs example template

Document the commonly used RUSTFS_* settings as commented examples in
the packaged conffile and point to docs.rustfs.com.
2026-09-20 11:27:41 +08:00
cxymds 22243e791d test(heal): cover stale current delete marker rejoin (#8021) 2026-09-19 19:33:24 +08:00
Hauser c35de00e8a fix(scanner): protect single-disk foreground latency (#8022)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-19 19:32:02 +08:00
cxymds 0e03bcda49 fix: serialize notify runtime module switch reconciliation (#8020) 2026-09-19 13:41:39 +08:00
cxymds c3269ad4c2 fix(heal): close tail-truncation and stale current-marker gaps (#8019)
* fix(heal): verify complete encoded shard tails

* fix(heal): prove stale current marker deletion
2026-09-19 13:41:30 +08:00
Chris 0fcd34c331 fix(ecstore): retry fleet capability probe while the notification system boots (#8017)
Startup finalizes IAM before init_notification_runtime publishes the notification system, and IAM finalization is what starts the fleet capability probe. The first probe pass therefore always failed closed and then slept the full ten-second probe interval. On a single node that left every durable capability, including the durable hard quota fence, unavailable for about ten seconds after /health already reported ok, so SetBucketQuota answered 503 durable quota capability is not confirmed across the cluster during that window.

Publishing the notification system now wakes the probe immediately through a Notify permit, with a 100ms bootstrap poll as the fallback for a wakeup that races the availability check. The probe keeps failing closed while the system is absent and logs the wait once. The four probe futures no longer carry an unreachable notification-system-unavailable arm, and the seven proof slots are revoked through one helper.

Fixes #8014
2026-09-19 13:41:16 +08:00
Chris ddc5f97a04 fix(release): sort pre-release packages below their final release (#8016)
Pre-release RPM versions replaced the SemVer `-` with `_`, which rpm treats as an ordinary segment separator, so `1.0.0_rc.5` compared as newer than `1.0.0` and a host with an rc RPM installed could not `dnf upgrade` to the GA package. DEB previews had the mirror-image problem: only the first `-` became `~`, so `1.0.0~rc.5-preview.2` carried `preview.2` as its Debian revision and sorted above `1.0.0~rc.5`.

Both formats now spell every pre-release separator as `~`, which dpkg and rpm (>= 4.10) both treat as "sorts before anything". Verified with rpm 4.16 (AlmaLinux 9) and dpkg 1.21: `1.0.0~rc.5 < 1.0.0`, `1.0.0~rc.5~preview.2 < 1.0.0~rc.5 < 1.0.0~rc.6`, alpha < beta < rc, and `rc.9 < rc.10`; fpm 1.18 passes `~` through into both package headers, and `dnf upgrade` from a `~rc.5` RPM to the GA RPM succeeds where the `_rc.5` one refused. The test now pins the full ordering contract under both package managers and fails against the previous spelling.

The release-checksum step already normalizes `~` to `.` for every uploaded asset, so RPM assets need no further handling. Hosts that already installed an `_rc`/`_beta` RPM need a one-time `dnf install rustfs-1.0.0` or `rpm -Uvh --oldpackage` to reach GA.

Fixes #8012
2026-09-19 03:22:44 +08:00
cxymds ff62c810b4 fix(tier): gate opaque version recovery cleanup (#8006)
* fix(tier): gate opaque version recovery cleanup

* test(tier): authorize opaque recovery fixtures

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
2026-09-18 21:54:03 +08:00
cxymds 950db7412c fix(heal): inspect stale members during deep scans (#8009)
* fix(heal): inspect stale members during deep scans

* test(heal): prove convergence outcomes
2026-09-18 21:53:31 +08:00
Chris 4e7a8fabb8 fix(ecstore): repair buckets left with only an incarnation sidecar (#8008)
The legacy bucket metadata migration writes the `.bucket-incarnation`
sidecar before `.metadata.bin`. A crash or a lost namespace lease between
the two writes left the bucket with a sidecar and no metadata, and every
later load failed closed with "bucket incarnation sidecar exists without
bucket metadata", so the bucket answered 500 to every request on every
node with no repair path (rustfs/rustfs#8003).

Load the sidecar-only state as a legacy bucket so the migration runs
again. The migration and the force-create path re-read the sidecar under
the transaction lock and keep its incarnation when persisting the
metadata, so the retry never replaces an identity other nodes fenced on.
A retired incarnation is never adopted: it is residue of a deleted
bucket, and re-publishing it would let heal reclaim the new objects.
2026-09-18 18:14:00 +08:00
cxymds b0f4e66c92 fix(tier): model R2 as unversioned storage (#8004) 2026-09-18 10:12:51 +00:00
cxymds e9da1fa074 feat: expose subscriptions used by event destinations (#8001)
* feat: expose event subscriptions by destination

* style: format event subscription handler

* fix: preserve s3 error footprint checks
2026-09-18 16:11:19 +08:00
Hauser 023adb1696 fix(heal): discharge absent durable partial writes (#8000)
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-18 16:07:18 +08:00
Hauser 0f69c073de fix(ecstore): preserve bounded listing completion reason (#7994)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-18 15:41:44 +08:00
cxymds 7a5e1efa95 fix(heal): fail admin heal with unhealthy drives (#7996) 2026-09-18 12:58:24 +08:00
Chris aa7f55946c docs(operations): add cluster and erasure-coding lifecycle runbook (#7998)
Consolidate the operator entry point for planning, storage classes and
EC:0 consequences, expansion, rebalance, decommission, heal, drive
replacement, restart recovery, and the rc CLI mapping. Every claim was
checked against the current handlers, storage-class validation, quorum
arithmetic, and the rustfs/cli admin reference; the runbook links the
normative contracts instead of restating them.

Refs rustfs/backlog#2607, rustfs/backlog#2608
2026-09-18 12:50:14 +08:00
Hauser 785b44a696 fix(replication): harden delete marker HEAD convergence (#7993)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-18 11:05:47 +08:00
Hauser 415bd2310c build(deps): bump dial9 to 0.5.1 (#7991)
Update dial9 and the resolved companion crates to 0.5.1.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-18 09:53:30 +08:00
Chris 3f469ab257 fix(helm): default the ingress backend port and refresh the nix cli hashes (#7990) 2026-09-18 09:08:24 +08:00
Hauser 31f446bed1 fix(ecstore): preserve vectored bitrot writes (#7981)
* fix(ecstore): preserve vectored bitrot writes

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): route test trait through contracts

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-18 08:07:37 +08:00
Hauser d2cdf057cc build(deps): refresh workspace dependencies (#7986) 2026-09-18 06:05:34 +08:00
Hauser 717e8b9c87 fix(scanner): resume and persist checkpoints across leader handoff (#7982) 2026-09-18 06:04:35 +08:00
Chris 5886368178 ci(performance): drop stale shared-lock comment from the performance workflow (#7987)
fix(ci): align security workflow test with the performance suite's own concurrency group

#7984 moved rustfs-performance-test.yml to the rustfs-performance-suite group but left test_all_suites_hold_the_shared_lock_for_manual_and_chain_runs asserting the shared functional group, so Quick Checks (make script-tests) fails on main and on every PR. Expect the dedicated group for the performance suite and drop the workflow comment that still described the shared lock.
2026-09-18 01:26:08 +08:00
cxymds db4b8290e7 fix(notify): validate bucket notification config before saving (#7980) 2026-09-17 22:58:29 +08:00
cxymds 5829c66f21 fix(decommission): keep scanner backlog handoff conflicts recoverable (#7979) 2026-09-17 21:41:11 +08:00
hector 9c199aefd1 ci(performance): move the performance suite to its own concurrency group (#7984)
The performance suite runs on the pf-testing fleet - a completely
separate environment (own runner, own target nodes) from the
smoke-testing fleet the functional suites share. Holding
rustfs-shared-functional-tests-v2 made an in-flight performance run
block every functional dispatch (observed: kms queued pending with
zero jobs for 4+ hours behind a 4-hour performance run), and
performance benchmarks queued behind functional suites in return.

Give it its own group; nothing else changes.
2026-09-17 20:43:55 +08:00
こころの在り処 4c02ad4025 feat(helm): add ingress support for both console and endpoint (#7972) 2026-09-17 18:30:29 +08:00
GatewayJ e31a22febb fix(ci): reject incomplete fault-tolerance runs (#7961)
* fix(ci): reject incomplete fault-tolerance runs

* fix(ci): validate fault-tolerance case verdicts
2026-09-17 18:20:33 +08:00
Hauser 91a6c5bad8 feat(release): add workspace cargo publish tooling (#7973)
* test(ecstore): bind MRF manifest CAS dirsync recovery (#7482)

Cover the committed MRF manifest path through LocalDisk conditional CAS when the metadata directory fsync fails. The fixture proves the previous manifest anchor survives rollback, an unanchored first successor is removed, and the legacy MRF journal remains readable even while global durability is relaxed.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): derive evidence runner profile from registry (#7484)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require mixed-version evidence roles (#7485)

Tighten the Scanner/Heal release bundle checker so mixed-version, scoped-ACK, and rollback fields cannot reuse a generic versions list without proving the expected evidence role.

Require version lists to use source revision identities and include the tested source revision. Also require profile evidence fields to name the core profiling artifacts before release approval.

The release gate remains blocked until measured field evidence is present.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): publish committed MRF runtime checkpoints (#7490)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): bind profile artifacts in release evidence (#7487)

Require Scanner/Heal release bundles to attach every required profiling artifact to P1 profile evidence with relative paths, artifact formats, non-empty files, hashes, and optional per-artifact measurement-window checks.

Document the tightened release bundle profile contract and cover missing, tampered, and mismatched-window profile artifact regressions in the existing checker self-test.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): harden measured ABBA evidence claims (#7491)

Reject measured Scanner/Heal ABBA manifests whose mixed-version evidence uses the same baseline and candidate source revision or binary hash.

Require crash fault modes and profile artifact names to match the exact supported sets, rejecting missing, duplicate, and unknown values.

Update harness fixtures and regression coverage for same-build mixed-version claims and exact-set release evidence fields.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(e2e): group scanner heal evidence payload (#7494)

Group the EC8+4 Scanner/Heal evidence writer inputs into a typed payload so the distributed e2e crate stays within the clippy argument limit without weakening the lint.

The evidence writer still validates the same S3 bodies, physical shard census, process restart PIDs, and node listings.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require MRF replay bundle fields (#7486)

Bind Scanner/Heal release bundle evidence for MRF durable replay to replay counts, retained responsibility anchors, and successor snapshot publication evidence.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): profile EC84 evidence case runs (#7495)

Give Scanner/Heal evidence cases explicit runtime profiles so EC8+4 background restart and crash cases use their own object count, object size, and partial-progress timeout defaults instead of inheriting the legacy 4x1 case assumptions.

Expose the runtime profile in plan-only output and cover every registry case in the script self-test.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): reject empty release evidence artifacts (#7496)

Require Scanner/Heal release bundle artifact paths to resolve to non-empty files before hashing them.

Cover empty hard-gate artifacts in the existing release bundle checker self-test and keep profile artifact size checking on the shared artifact boundary.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require G14 same-window field coverage (#7489)

Tighten the Scanner/Heal release bundle checker so G14 same-window evidence must name the EC8+4, multi-set, and multi-pool fields covered in that measurement window.

Keep the release gate blocked when same-window evidence omits one of the required G14 fields, without changing production runtime behavior.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require two-hour measured ABBA windows (#7493)

Reject measured Scanner/Heal release ABBA manifests and summaries whose evidence window is shorter than the W21 two-hour release requirement.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): cover MRF idle checkpoint cleanup (#7497)

Add a runtime cleanup regression test that publishes both retained replay and runtime committed checkpoints, writes scoped and legacy journals, and verifies idle cleanup removes every recovery anchor from the registered local disks.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(e2e): ignore untracked files in build identity (#7498)

Align e2e_test build provenance with Scanner/Heal evidence receipts and server binary provenance by treating only tracked source changes as dirty.

This prevents unrelated untracked worktrees or evidence directories from causing compiled test identity mismatches before real evidence cases can run.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require MRF disk-full evidence fields (#7499)

Require the Scanner/Heal release registry and bundle checker to carry explicit G08 MRF capacity, disk-full, and replica-loss evidence fields before release approval can pass.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): bind release JSON artifact provenance (#7500)

Require scanner/heal release evidence JSON artifacts to repeat their measured source revision, run identity, measurement window, gate, and field identity inside the artifact payload. This keeps a refreshed outer bundle hash from accepting stale summary or profile JSON from another run.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): cover MRF rollback mirror filtering (#7501)

Add a regression oracle that keeps scoped-only MRF responsibilities in the authoritative runtime snapshot while omitting them from the v1 legacy rollback mirror.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require segment activation evidence (#7504)

* test(scanner): require segment activation evidence

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): cover activation proof inputs

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require MRF crash matrix cases (#7505)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require release bundle domain evidence (#7502)

Reject scanner/heal release bundles that omit field-level domain evidence for scoped ACK, durable intent, mixed-version, scheduler pressure, profile cost, and two-hour pressure lanes.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require MRF retention GC evidence (#7506)

Require P4 retained-responsibility release bundle evidence to list the retained replay anchor and idle cleanup cases, prove a two-hour retention window, and record both idle cleanup and verified-proof discharge observations.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require MRF cleanup GC soak evidence (#7507)

Require the P4 release bundle to carry measured cleanup/GC soak evidence for retained MRF replay responsibilities. The bundle now needs a two-hour cleanup window, exact cleanup case coverage, observed verified idle GC, and zero pending responsibilities or stale journals after GC.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): cover MRF disk-full commit anchors (#7509)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(e2e): report port allocator bind errors (#7510)

Surface the scanned port window, attempt count, and last bind error when the e2e port allocator cannot reserve a localhost port. This keeps Scanner/Heal evidence failures actionable when the environment blocks binds before business assertions run.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(sse): resolve 1.0.0 SSE/KMS blockers and P1 findings (#7511)

* fix(sse): resolve bucket default encryption per request

PUT and the POST-object/extract path resolved a bucket's default
encryption with a hard-coded "no explicit SSE-C" flag, so the default was
layered onto a request that already carried an SSE-C header triple and
then tripped that request's own mutual-exclusion check. Every bucket with
default encryption refused SSE-C single PUTs with 400 InvalidArgument,
while CreateMultipartUpload on the same bucket succeeded because it
resolves SSE elsewhere. Both call sites now derive the flag from the
request headers, as COPY already did.

The bucket default's KMS key id was also inherited independently of the
effective algorithm, so an explicit AES256 request against an aws:kms
default bucket produced a self-contradictory algorithm/key-id pair and
was rejected. The key id is now inherited only when the effective
algorithm is aws:kms, matching the storage-layer resolver.

Refs backlog#2368 B1, B2.

* fix(sse): refuse SSE-KMS without a running KMS service

A write requesting aws:kms on a node with no KMS service fell back to the
node-local SSE-S3 provider: the data key was wrapped with
RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded
aws:kms and the requested KMS key id. The stored object claimed a KMS
protection it never had, under a key that was never consulted, and no
signal distinguished it from a genuine SSE-KMS object.

The managed-encryption path now asks the resolved DEK provider whether it
wraps with a node-local master key and refuses SSE-KMS in that case:
InvalidRequest when KMS was never configured, ServiceUnavailable when a
configured service is not running. The check sits after the per-key
authorization gate so an unauthorized caller still receives AccessDenied
whatever the KMS runtime state is, and asks the provider rather than a
parallel availability signal because the provider is what actually wraps
the key. A missing master key no longer answers an SSE-KMS request with
an SSE-S3-worded configuration error.

The SSE-S3 local fallback is unchanged.

Refs backlog#2368 B4.

* fix(ecstore): restore and archive tiers in stored coordinates

Multipart restore addressed the remote tier in plaintext coordinates
while the copy-back reads the stored representation. Each part received a
misaligned slice of the remote object whose length still satisfied the
range, the hash reader and the completion size check, so the restore
reported success and silently replaced the object's bytes. Encrypted and
compressed multipart objects were both affected. Restore now accumulates
stored part sizes, passes the stored length to the hash reader alongside
the plaintext length, and validates against the stored size.

The copy-back digests stored bytes, so its computed MD5 is not the
object's public ETag. Restore now preserves the object ETag on both the
single-part and multipart paths, and gives each restored part its own
recorded part ETag rather than the object-level value.

Transition also handed the tier the object's SSE headers and its
RustFS-wrapped data key as request headers. Any S3 target rejected an
SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a
second time under a key id it does not own, and the wrapped DEK left the
cluster. The archive request now strips every SSE header and encryption
marker with the predicate the replication path already uses; the local
xl.meta keeps all of it, so read-through and restore are unaffected.

Objects restored by an affected release are not detected or repaired
retroactively and must be re-restored from the tier.

Refs backlog#2368 B3, B5; backlog#2369 P7.1.

* fix(rio): lock the v1 nonce layout within a segment

Decrypting a v1 segment tried three historical nonce layouts per frame,
independently for every frame. The last of them exists for streams
written before 1.0.0-alpha.91, which reused a segment's part nonce for
every block in it; because block zero's derived nonce equals that base
nonce, a frame encrypted at index zero authenticated at any position. An
attacker able to rewrite the underlying shards could replay it and have
the forged plaintext returned with 200 and an unchanged length. Shard
integrity uses a keyed-hash-free checksum, which such an attacker can
recompute, so it is not a barrier.

A segment now locks onto whichever layout decoded its first non-zero-index
frame and rejects any later frame needing a different one. That leaves one
residual shape: a stream built purely from repeats of frame zero has no
later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK
(default true, so pre-alpha.91 objects keep decrypting) drops the third
layout entirely when set to false, which closes it. Turning it off refuses
pre-alpha.91 objects, so migrate them first by rewriting in place.

Refs backlog#2369 P2.

* fix(kms): reload a service that failed to start

POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted
configuration matched the in-memory one byte for byte. A node whose KMS
failed to start keeps that configuration and sits in Error, so the
documented recovery call returned "reloaded successfully" while leaving
the node down. Peers reached the same path through the reload broadcast,
so a cluster that lost Vault during a rolling restart had no working
recovery route other than the node-local start endpoint. Reload now
short-circuits only for a service that is actually running, and otherwise
reconfigures, which starts a service that is not running.

The AWS backend also advertised key-version enumeration through
kms/status, which its own documentation says it cannot do; the capability
and its golden snapshot now say false.

Refs backlog#2369 P1, P7.3.

* docs: record the SSE and KMS changes for 1.0.0

The Unreleased changelog section carried no entry for any encryption work
merged since 1.0.0-rc.5, including three items with operational impact:
the config-secret variable whose absence persists secrets in cleartext
with only a warning, the v2 frame write switch and its rolling-upgrade
constraint, and per-key authorization making a public bucket incompatible
with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS
refusal as a breaking change with both routes out.

Also corrects four places where documentation contradicted the code: the
cleanup register still called encrypted range seek opt-in after its
default flipped, the Helm README claimed vault_mount_path only applies to
Transit while the template also feeds the KV2 mount, the disaster-recovery
drill listed bundle contents for backends whose export is refused with
501, and the Chinese README capability table predated most of the feature
set. Documents the SSE-S3 local master key as a first-class operational
mode with its rotation dead end, and what the v1 frame layout does and
does not authenticate.

Refs backlog#2369 P5.

* fix(kms): classify data-path KMS failures by what the caller can do

Only "key not found" and a backend outage were classified; every other
KMS failure that reached the S3 data path fell through to
500 InternalError with a generic message. A disabled or pending-deletion
key, a denied KMS grant, an encryption-context mismatch, an unsupported
algorithm, a credential or timeout failure, and a capability the
configured backend does not have all looked identical to a server fault.
SDKs therefore applied exponential backoff to configuration errors no
retry can fix, and monitoring counted every one of them against the
server's own error rate.

Unusable-key and request-side failures now answer 400, a denied grant
403, transient backend failures 503, and a missing backend capability
501. Damaged, unreadable, or unknown-format key material keeps its 500:
it is a server-side integrity fault, and existing tests pin it.

The classifier is deliberately separate from the admin lifecycle
mapping, which answers 404 for a missing key because there a key id is
the resource being addressed; on the data path it arrives inside a
request header or a bucket default. Messages either name what the caller
asked for or stay generic, with deployment-side detail left on the error
source the way the storage-IO mapping already does.

Refs backlog#2368 B6.

* fix(kms): track and renew static Vault tokens

Token authentication hard-coded "this token carries no lease", so the
renewal task never started, the remaining-TTL gauge was never published,
and nothing looked wrong. `vault token create` grants a 768-hour TTL by
default, so a cluster that had been healthy for a month turned every KMS
call into a 403 and could not recover without a restart or a
reconfigure. Production configuration validation only rejects the
literal dev-token, so an ordinary expiring token reaches a whole cluster.

The source now reads `auth/token/lookup-self` at login and adopts what
Vault reports. A token with no expiry behaves exactly as before. An
expiring renewable one is picked up by the existing renewal loop and
renewed at half TTL like every other auth method. An expiring
non-renewable one warns with its remaining lifetime and publishes the
gauge, so the fail-closed window is visible before it arrives.

The probe never fails the login: a policy that omits lookup-self, or a
Vault that is briefly unreachable, warns and falls back to exactly the
previous behaviour rather than taking down a deployment that works
today. The scripted Vault test double answers the lookup out of band so
existing scripts keep describing only the protocol under test.

Refs backlog#2369 P3.

* feat(sse): report SSE-C requests that arrive without TLS

An SSE-C request carries the customer's AES key in a request header, so
AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS
accepted them on any transport: a plaintext hop hands the key to anyone
on the path, and since the object cannot be read without that same key,
the exposure lasts as long as the object does.

Refusing outright is the correct end state but not a safe default to
adopt inside a release window, because the project's own s3-tests and
e2e lanes and most staging deployments speak plain HTTP. This release
reports instead: each such request increments
rustfs_ssec_plaintext_requests_total and logs one warning per process, so
an operator can confirm nothing would break before the default flips.
RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now.

The verdict is per connection rather than per deployment: the layer is
built with whether this listener terminated TLS, and additionally accepts
an https protocol forwarded by a proxy the trusted-proxy configuration
already vetted. It sits beside the rate limiter, after the layer that
makes a forwarded protocol trustworthy and after the request context, so
a rejection can echo the request id.

Refs backlog#2369 P7.2.

* fix(kms): say what a node-local backend means for a cluster

The Local backend keeps key material on each node's own disk and
generates its Argon2id salt per node, so two nodes derive different keys
from the same master_key and an object encrypted on one node cannot be
decrypted on another. Behind a load balancer that surfaces as
intermittent 500s on reads that succeeded moments earlier, with nothing
tying the symptom to the cause: the only signal was a generic
"development, testing and demos only" positioning warning that says
nothing about what actually breaks.

Configuring or reconfiguring Local while the deployment is distributed
now logs a dedicated event and appends the consequence to the configure
response, so the operator who made the change sees it. The product
decision to warn rather than refuse is unchanged.

Refs backlog#2369 P7.4.

* docs: record the remaining SSE and KMS changes for 1.0.0

Adds changelog entries for the KMS data-path status classification, the
Vault static-token lease probe, the SSE-C plaintext-transport report and
its switch, and the node-local backend warning.

Documents two things the backend security guide never stated: that SSE-C
belongs on a secure transport, with the counter and switch to plan the
change around, and that the Local backend cannot be shared by a
multi-node deployment because each node derives different keys from the
same master key.

Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4.

* fix(kms): report an unreadable key store as an outage on the S3 path

A backend now distinguishes a key store it could not read from a key
that is genuinely absent, but the S3 boundary collapsed the first one
back onto 500 InternalError through the fallthrough for integrity
faults. The distinction was therefore invisible to the client: a
temporary key-directory outage looked exactly like a permanently damaged
key record, and neither the status nor the metric said the request was
worth retrying.

An unreadable key store joins the retryable class and answers 503, next
to a backend error and a credential failure. Damaged, unreadable or
unknown-format key material keeps its 500.

Refs backlog#2368 B6; builds on rustfs/rustfs#7470.

* test(storage): cover MRF cleanup delete ENOSPC anchor (#7513)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require heal retry stats evidence (#7514)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add release bundle dry-run fixture (#7515)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): confirm recovery intent accept readback (#7516)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require MRF cleanup delete ENOSPC evidence (#7518)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require bounded retry window evidence (#7517)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(e2e): emit scanner heal G09 upgrade evidence (#7519)

Record measured Scanner/Heal G09 evidence artifacts from the upgrade compatibility lanes when a fresh evidence directory is provided.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): persist MRF rollback mirror boundary (#7520)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add G09 upgrade evidence runner (#7522)

* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* test(scanner): add G09 upgrade evidence runner

Add a reusable Linux x86_64 runner for the Scanner/Heal G09 mixed-version and rollback upgrade evidence lanes.

The helper reads the pinned previous-release asset metadata from the upgrade workflow, verifies the downloaded binary, builds the current head, runs both ignored E2E tests, and fails unless the expected G09 JSON artifacts exist.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): implement V2 evidence preflights (#7523)

* feat(scanner): wire dirty usage producer identities

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add segment activation preflight proof

Keep scanner segment reuse behind a structured activation preflight so release evidence can prove the production gate remains disabled until every producer, generation, overflow, cold-oracle, and distributed invalidation check is satisfied.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): expose distributed invalidation evidence

Record an explicit distributed segment invalidation evidence summary when remote dirty usage snapshots are bound to the current activity window and the authenticated scoped ACK capability probe succeeds.

Reject peer dirty usage snapshots that contradict the peer activity pending bit so scoped ACKs fail closed instead of clearing an unadvertised remote mutation.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): assemble release evidence bundles (#7526)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add G09 upgrade evidence runner (#7524)

* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* test(scanner): add G09 upgrade evidence runner

Add a reusable Scanner/Heal G09 runner for Linux PR-head validation. The script downloads the pinned previous release, builds the current checkout, runs the mixed-version and rollback upgrade E2E lanes, and validates the measured raw artifacts before they can be consumed by the release bundle gate.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): replay recovery intents while disabled (#7521)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): preflight G09 evidence disk space (#7527)

* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* fix(targets): reject trailing batch items (#7508)

* test(scanner): preflight G09 evidence disk space

Fail the Scanner/Heal G09 upgrade evidence runner before downloading or building when the validation host does not have enough free space for a full raw evidence pass.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): respect cargo target dir in G09 runner (#7528)

* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* fix(targets): reject trailing batch items (#7508)

* fix(scanner): respect cargo target dir in G09 runner

Write the RustFS feature stamp under the resolved Cargo target directory so remote validation hosts with CARGO_TARGET_DIR set can reuse the built PR-head binary.

Extend the runner plan/self-test path to cover relative target-dir resolution.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(e2e): add G09 upgrade evidence runner (#7525)

* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* test(e2e): add G09 upgrade evidence runner

Add a Linux x86_64 runner that downloads the pinned previous release, builds the current RustFS binary, runs the mixed-version and rollback upgrade compatibility lanes, and verifies the required Scanner/Heal G09 raw evidence artifacts.

Document the runner and add a shell self-test for help, dry-run, SHA validation, and non-empty artifact directory guards.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): emit G09 release bundle gate evidence (#7530)

* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* fix(targets): reject trailing batch items (#7508)

* test(scanner): emit G09 release bundle gate evidence

Write a bundle-ready G09 gate descriptor from the Linux upgrade evidence runner and validate the single G09 gate with the shared release-bundle rules without approving the full Scanner/Heal release.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): reject invalid G09 runner test selection (#7531)

Validate the selected G09 evidence lane before process-substitution case expansion so --plan-only cannot turn an unknown --test value into an empty successful plan.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): keep G09 source binary path clean (#7532)

Keep checksum verification output out of the command substitution that resolves the previous-release binary for the G09 runner.

Also tolerate non-GNU sha256sum in local self-tests by falling back to shasum when GNU --check support is unavailable.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): cover MRF replay proof cleanup (#7534)

Exercise the committed MRF replay checkpoint path with a real ECStore-backed bucket incarnation, retain the replay anchor until an exact verified proof arrives, then delete the proof-discharged checkpoint and prove a subsequent restart does not resurrect the replayed work.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): carry segment activation preflight evidence (#7535)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): surface cold segment reuse oracle (#7536)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require hard evidence artifact payloads (#7537)

Require Scanner/Heal release bundle JSON artifacts to carry hard-domain evidence fields for mixed-version, crash, capacity, disk-full, replica-loss, and MRF cleanup gates. This prevents a descriptor from approving a hard gate while pointing at a generic measured artifact that lacks the boundary-specific oracle fields.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): bind producer evidence to dirty generations (#7538)

Record scanner segment producer identities with the dirty usage generation they invalidated, then expose a cycle-local producer evidence snapshot for segment reuse activation preflight.

Production activation remains fail-closed until durable producer identity and restart-gap proof are available.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): persist segment invalidation proof metadata (#7539)

* test(scanner): wait for EC8+4 heal control readiness (#7540)

* test(scanner): list full P4 release evidence fields (#7542)

Align the Scanner/Heal release requirement registry with the release bundle gate for P4 so the closure checklist advertises MRF scale, replay cost, retained responsibility, and cleanup/GC soak evidence together.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add W16 release evidence runner (#7543)

Add a W16 Scanner/Heal evidence runner that executes the recovery-intent crash-boundary and quota-authority lanes, emits measured G04/G12 JSON artifacts, and validates the resulting single-gate release descriptors.

Wire its shell self-test into script-tests and document the release evidence entry point.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix: address release acceptance regressions (#7541)

* feat(scanner): admit durable segment proof replay (#7544)

* feat(scanner): admit durable segment proof evidence

Validate complete set snapshot segment invalidation proof metadata against the current dirty usage generation window and scanner process epoch before clearing durable producer and restart-gap activation blockers.

Production segment reuse remains gated by the explicit activation flag.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): replay segment proof from root snapshots

Carry segment invalidation proof metadata into root snapshot set states so a complete published baseline can replay the durable producer evidence recorded by each set cache.

Keep the field additive for older readers and leave incomplete or LKG set states unproven.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): bind MRF release artifacts to evidence kinds (#7545)

Require Scanner/Heal G07/G08/P4 JSON artifacts to declare the exact MRF evidence kind expected for each release-bundle field. This keeps generic measured JSON from satisfying durable replay, ENOSPC, replica-loss, scale, replay-cost, retained-responsibility, or cleanup-GC soak gates.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): enable segment reuse activation gate (#7546)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): run scoped fallback cases on dedicated stack (#7550)

Linux libtest defaults can abort these deep EC-store futures before the assertions run. Wrap the async scoped fallback cases in the repository's dedicated-stack test pattern so the default cargo test path exercises the oracle directly.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): roll out distributed e2e selection (#7549)

Refresh the committed e2e-distributed selection digest after the EC8+4 Scanner/Heal evidence case joined the profile, and document the registry-aware runner entry for the distributed case.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add W13 MRF evidence runner (#7547)

Add a W13 durable MRF evidence runner that emits measured G07, G08, and P4 JSON artifacts and release descriptors through the existing scanner/heal bundle gate.

The runner now executes the ignored MRF replay evidence test with an exact full test path, validates raw artifact kinds and gate decisions, prepares Linux tmpfs-backed ENOSPC roots for G08, and documents the Linux/long-soak boundaries.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): fix segment proof fixtures

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): reduce W13 evidence helper arguments

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* ci(test): enable local KMS for s3 compatibility tests (#7553)

Configure the s3-tests harness with a local KMS key so SSE-KMS cases run in CI without relying on an external KMS service.

Also move the anonymous POST default SSE-KMS regression onto the shared local KMS test environment.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): bind segment proof to typed dirty suffix (#7564)

Track producer identity completeness per pending bucket and retain process coverage across durable acknowledgements. Admit a prior same-process segment proof only for an exact or strictly earlier fully typed generation window.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): validate scoped ACK clear receipts (#7563)

Fail closed when a signed scoped ACK response claims more cleared entries than its request contained, while preserving partial-clear semantics and zero-mutation capability probes.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): bind cold scoped cache reuse (#7565)

Record an explicit cold bucket reuse proof for scoped set scans and treat fully cold sets as valid cold-zero-walk reuse when the baseline and bucket incarnations are bound.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(s3): preserve delete marker read error headers (#7558)

* chore(release): sync configurable nightly build workflow from main (#7567)

feat(nightly): make the build branch configurable (NIGHTLY_BRANCH var + dispatch input) (#7557)

* docs(obs): correct metric export interval and explain alert timing (#7571)

* fix(heal): persist scoped admin admission handoff (#7569)

* fix(heal): persist root admission handoff

Persist administrator root heal ownership before queue publication so crash restart can replay an admitted control-plane start without relying on process-local replay cache state.

Protect queued durable root owners from priority displacement; callers must use forceStart cancellation semantics to replace them.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): persist admin heal handoff types

Extend durable admin heal admission handoff beyond cluster roots while preserving legacy root recovery records. Persist the heal type in schema v2, replay typed admin requests after restart, keep non-admin requests out of the durable control-plane handoff, and allow path cancellation to clear durable-only non-root records.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): preserve disjoint durable admin owners

Limit admin forceStart durable-owner cleanup to overlapping heal requests after the admin recovery journal begins storing non-root work. Add fail-closed and displacement tests so durable bucket owners cannot be silently dropped by unrelated replacement or priority pressure.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): persist admin terminal receipts

Persist terminal receipts for durable administrator heal starts before retiring their pending intent so restart can answer completed or cancelled scoped tasks without replaying them.

Retain pending work when terminal publication fails and keep timeout owners on checkpoint-only replay semantics.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): mark committed replication and tier mutations (#7568)

* fix(scanner): mark committed replication and tier mutations

Bridge ECStore committed replication status writebacks and lifecycle tier expiration cleanup into scanner dirty-usage producer identity tracking. Only successful replication metadata updates mark Replication; validate-only, superseded, and retry paths remain unmarked.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): replay durable dirty producer records

Add a strict scanner dirty-usage producer replay schema that can restore typed producer coverage after restart without treating ordinary process-local mutations as durable evidence. Replay records validate schema, cache key format, generations, scopes, producer identities, duplicate buckets, byte limits, and entry limits before modifying dirty state.

Keep segment activation fail-closed unless the restored bucket state came from durable replay; subsequent local mutations clear the durable authority bit for that bucket. Also fix the release scanner clippy lint in scoped cold-reuse proof creation.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): recover expansion buckets and resume retirement (#7574)

* fix(ecstore): recover partially created expansion bucket volumes

* fix(ecstore): resume retirement after startup safety checks

* fix(ci): align S3 KMS fixtures and default key expectations (#7573)

* fix(heal): prune retained root terminal receipts (#7576)

Bound durable root-heal terminal receipt retention to the completed-status TTL while keeping expired receipts authoritative enough to retire stale pending intents before deleting the receipt itself.

Add a budgeted background GC pass and fail closed on corrupt or non-terminal receipt records.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(storage): preserve retryable publication guard errors (#7561)

* fix(storage): preserve retryable publication guard errors

* fix(scanner): satisfy scoped cache clippy lint

Use bool::then_some for the scoped cold bucket reuse proof construction after merging the latest release branch.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(health): report storage quorum in node readiness (#7566)

* fix: preserve scanner backlog during pool retirement (#7555)

* fix: hand off native scanner backlog before pool retirement

* test: initialize optional proof in data usage fixtures

* perf: bound parallel V3 pool metadata phase writes

* test: keep native backlog fault errors typed

* test: skip unchanged pool metadata in crash barriers

---------

Co-authored-by: cxymds <cxymds@gmail.com>

* test(scanner): bind profile artifact provenance (#7577)

* fix(scanner): persist dirty producer journal (#7580)

Persist scanner dirty producer replay records from committed replication and tier mutations, hydrate journal state on restart, and clear durable records only after acknowledged dirty generations.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test: add scanner heal scheduler pressure evidence (#7584)

Add measured G10/P1/P3 scheduler-pressure release descriptor generation, ABBA scheduler/profile provenance, and focused self-tests for the release bundle gate.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test: add scanner heal G14 multiset evidence (#7583)

Add EC8+4 multi-set and multi-pool scanner/heal release evidence coverage, including e2e registry cases, oracle checks, and a G14 descriptor assembler for measured case artifacts.

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(nightly): push assets with plain git on release — the build fleet has no gh CLI (#7586)

fix(nightly): push assets with plain git — the build fleet has no gh CLI

The sm-standard-4 runners used by the nightly build have no gh binary
(functional-chain workflows run elsewhere, on the jumpbox). The assets
publish step died on 'gh: command not found' at the credential-helper
setup, and the preceding clone failure had been masked by 2>/dev/null,
misleading the step into the orphan path. Swap clone and remote setup
to plain git with the token embedded in the URL; push semantics are
unchanged.

* fix(scanner): stabilize retired backlog survivor commits (#7585)

Treat an authoritative scanner pause-backlog commit as stable when every member in that commit stores the matching stable and committed record. Retired source replicas can remain missing in the observed topology while survivor membership is already stable.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(metrics): prevent duplicate drive reporting across nodes (#7582)

Co-authored-by: houseme <housemecn@gmail.com>

* fix(ecstore): avoid Windows batch-delete cleanup sharing conflicts (#7579)

Co-authored-by: houseme <housemecn@gmail.com>

* feat(ilm): inspect legacy transition version state (#7581)

* test(scanner): add release evidence ingestion gates (#7588)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix: reuse admin error boundary for legacy tier reconciliation (#7587)

* fix(test): stabilize release CI heal fixtures (#7589)

Keep heal admission-only tests off durable root recovery so mock storage does not require local metadata disks. Align forceStart retry assertions with replacement semantics and fix the e2e metric clippy collapse.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)

Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).

- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
  the 'nightly' release via the REST API, deletes same-name assets,
  uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
  body with the build provenance (ref@sha, run link, sizes, SHA256).
  Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
  exercised end-to-end against the real release with probe files
  (create / upload / overwrite / download round-trip / body update).

* test(scanner): add release bundle evidence descriptors (#7595)

* test(scanner): merge release evidence descriptors

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add authority and legacy release descriptors

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add scoped ack release descriptor

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require all release evidence fields

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add mrf release descriptor producer

Add a standalone W13 MRF descriptor producer for measured G07/G08/P4 raw artifacts, with fail-closed self-tests for missing responsibility cases, disk-full ENOSPC observations, synthetic artifacts, and short P4 cleanup soaks.

Also bind G12 quota path evidence to explicit case lists so W16 descriptors cannot pass without reset and settlement coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): reuse mrf release descriptor producer

Route the W13 runner through the shared MRF descriptor producer so measured runs and offline re-packaging use the same fail-closed validation.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(ilm): backfill proven legacy transition state (#7591)

* feat(ilm): backfill proven legacy transition state

* fix(ilm): remove redundant reconcile test clone

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): remove redundant transition clone (#7597)

Drop an unnecessary clone in the legacy transition state digest regression so release CI clippy lanes can run with warnings denied.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): seed current backlog members before committing (#7594)

* fix(scanner): seed current backlog members before committing

* test(app): run recursive force delete test on large stack

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): bind raw profile artifacts (#7598)

Require Scanner/Heal release profile wrappers to carry bundled raw profile artifacts plus the matching profile cost metrics.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): surface first failed object in status (#7599)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): cover retry budget across attempts (#7600)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): harden durable producer evidence gate (#7601)

Require Scanner/Heal G11 maintenance evidence to prove durable dirty producer journal replay, replay-state hydration, generation-bound ACK clearing, and invalid journal fail-closed cases before segment reuse can be activated.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): harden scoped ack evidence proof (#7602)

Require Scanner/Heal G03 scoped ACK evidence fields to carry their own measured provenance and concrete ACK, capability, and mixed-peer observations before release-bundle gate verification.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): bind status outcome raw provenance (#7603)

Require measured G05/G06/R-D raw status, compatibility, and disposition artifacts to share run identity before producing release bundle descriptors.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): retry fully recoverable batch failures (#7604)

Allow recursive bucket and cluster heal attempts whose retained batch failure is entirely retryable to use the existing bounded manager retry budget. Permanent or mixed batch failures remain terminal so heal outcome semantics stay strict.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): avoid movement debt for stale remote lease (#7605)

Classify a final remote scanner publication lease validation failure from a fresh activity snapshot instead of treating every validation error as data movement. If the peer session and movement generation still match the granted leases and publication is otherwise allowed, keep the publication rejected as an activity-baseline miss without creating pause backlog movement debt. Preserve DataMovement for expired leases, peer restarts, movement generation changes, and active publication blocks.

Tests cover namespace-only validation invalidation and the remote fence-loss cases that must still defer as DataMovement.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* chore(deps): refresh workspace dependencies (#7606)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): bind G14 case artifacts (#7611)

Require Scanner/Heal G14 release bundle JSON wrappers to mirror their outer evidence and carry self-contained case artifact provenance.

Copy proof-json case artifacts into the generated G14 descriptor bundle so assembled release bundles can validate case file hashes after relocation.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): retry contended decommission cancel target locks (#7612)

* test: align transition fixtures and KMS context expectations (#7610)

* test(scanner): stabilize W13 release gate evidence (#7613)

* feat(nightly): publish packages as assets of the rolling 'nightly' release (sync from release) (#7593)

feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)

Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).

- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
  the 'nightly' release via the REST API, deletes same-name assets,
  uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
  body with the build provenance (ref@sha, run link, sizes, SHA256).
  Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
  exercised end-to-end against the real release with probe files
  (create / upload / overwrite / download round-trip / body update).

* test(scanner): stabilize W13 release gate evidence

Treat only real raw-entry windows as replayed scanner enumeration in the restart diagnostic, so final completion rounds without raw entries are not fail-closed as raw replays.

Allow the EC8:4 multi-pool heal evidence case to select an outage object key that routes to an online pool while an entire target pool is down, preserving strict behavior for single-pool cases.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): bound native backlog fixture resources (#7609)

* fix(scanner): bound native backlog fixture resources

Keep native scanner backlog restart fixtures within low file descriptor limits by reducing the native-only disk layout, making fixture shutdown drain background work, and avoiding long-lived ECStore retention from background loops.

Convert ECStore-backed background refresh/recovery/monitor tasks to upgrade weak owners only while doing work so completed test stores release their disk graph before the next native fixture starts.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(tier): refresh release-merged failure fixtures

Reuse the raw legacy-transition fixture helper so delete-all expiry actually exercises Unknown transitioned history metadata after the release merge.

Rewrite restore-failure disk fixtures through per-disk xl.meta snapshots so stale destination identities can be persisted without tripping ordinary metadata update guards.

Update the SSE KMS mismatch expectation to the InvalidRequest/context_mismatch behavior now returned by the rio-v2 path.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ci): align release E2E gate and repair regression tests (#7607)

* fix(scanner): harden heal release gates (#7614)

* test(scanner): reject synthetic heal evidence inputs (#7617)

Harden the Scanner/Heal checkpoint restart and G14 EC evidence producers so release descriptors cannot be assembled from marked fixture, dry-run, or synthetic case inputs.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): collect status outcome raw evidence (#7618)

Normalize live Scanner/Heal status/outcome observations into the measured raw artifacts consumed by the G05/G06/R-D release descriptor producer.

Reject synthetic observations, incomplete required cases, and reused run/window identities before raw artifacts can enter the release bundle flow.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): bind scoped ack confirmation to generation (#7619)

Require lost scoped dirty-usage ACK reconciliation to observe a clean peer activity generation that covers the requested ACK generation. Apply the same guard in the scanner aggregation path so a stale clean activity snapshot cannot discharge pending maintenance after an uncertain ACK response.

Refs rustfs/backlog#2427

Refs rustfs/backlog#2281

Refs rustfs/backlog#2240

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test: report tier reference proof setup failures without hanging (#7608)

* test(scanner): plan Linux evidence validation (#7620)

Add a Scanner/Heal Linux release-evidence planner that emits a machine-readable execution manifest for the remaining measured validation lanes.

The planner can run only lightweight preflight checks and keeps plan-only output distinct from measured release evidence.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(e2e): select G14 replacement drive by evidence (#7622)

Choose the replacement disk from the target node by the presence of complete pool metadata instead of assuming the first configured drive is the scanner metadata holder. This keeps the G14 multi-set harness aligned with multi-drive EC layouts and adds clearer diagnostics when multi-pool outage writes fail closed.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): report Linux evidence plan status (#7624)

Add a status mode for the Scanner/Heal Linux evidence planner so operators can identify missing or incomplete artifacts before final bundle assembly.

The status output stays plan-only and never reports release approval.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add G14 multi-pool background heal evidence (#7628)

test(scanner): align G14 multi-pool outage evidence

Treat the localhost multi-pool topology as whole-pool loss when the target node is down. If that topology cannot admit the outage object while the pool is offline, defer that object write until the pool rejoins and record the oracle marker instead of failing before the real crash/restart evidence runs.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): verify deferred multipool outage evidence (#7629)

Validate the G14 multi-pool oracle field that records down-window outage PUT refusal and requires the deferred post-rejoin outage object to be accepted and checked through S3.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): allow metadata snapshots at read quorum (#7627)

fix(ecstore): admit metadata snapshots at read quorum

Allow guarded bucket metadata snapshot existence checks to use read quorum so degraded erasure sets can continue Object Lock snapshot reads without weakening bucket mutation or object write quorum.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* chore(deps): refresh release dependencies (#7632)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): use pool lock quorum for set writes (#7630)

Use the pool-wide namespace lock client domain for every set in a pool so degraded EC8+4 multi-set writes are gated by node-level lock quorum instead of the narrower per-set endpoint host slice.

Keep namespace-lock domain deduplication tied to both the pool namespace and shared clients, and add regression coverage for three-locker degraded writes plus cross-pool domain separation.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(e2e): keep G14 multi-set restart graceful (#7633)

test(e2e): keep multi-set restart graceful

Include the EC8+4 multi-set restart scenario in the graceful interruption lane so the unclean-shutdown marker assertion matches the scenario semantics.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(e2e): record deferred G14 outage writes (#7637)

Include optional outage-write diagnostics in G14 evidence oracles so deferred multi-pool outage writes bind their down-window refusals and post-rejoin acceptance.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): bound G14 multi-pool evidence size (#7640)

Keep the multi-pool evidence runner within the registry object budget while retaining deferred outage-write diagnostics for release-gate validation.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): find nextest junit fallback for evidence cases (#7643)

Handle cargo-nextest writing JUnit reports under the workspace target directory even when the Rust build uses CARGO_TARGET_DIR. This keeps measured Scanner/Heal evidence cases from passing the real test but failing final receipt packaging.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ilm): preserve cleanup ownership on tiered overwrites (#7639)

* fix(ilm): preserve cleanup ownership on tiered overwrites

* fix(ci): refresh E2E selection for tier overwrite regression

* test(ilm): restore noncurrent compensation tests to serial CI (#7645)

* fix(heal): preserve null-version tags and encoded object paths (#7644)

* fix(e2e): record EC8+4 drive restart set size (#7648)

* fix(e2e): record EC8+4 drive restart set size

Include the erasure set drive count in the distributed EC8+4 drive restart oracle so Scanner/Heal release evidence matches the registry contract.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): tighten s3s footprint ratchet

Replace release-merge s3_error! macro calls with equivalent S3Error constructors so the s3gate migration ratchet does not grow on the PR merge tree.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(e2e): target multi-set outage heal candidate (#7653)

* test(e2e): target multi-set outage heal candidate

Require the outage write used by EC8+4 multi-set root-heal evidence to miss the same erasure index owned by the selected replacement drive. This avoids accepting a candidate from a different set and turning a valid heal into a false negative.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(e2e): satisfy G14 heal lint gates

Remove clippy-only noise from the G14 multi-set heal evidence test and align the admin route policy inventory with the registered heal catch-all route.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* update

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): reuse API error boundary for decoding failures (#7647)

* fix(heal): reuse API error boundary for decoding failures

* Update heal.rs

Signed-off-by: houseme <housemecn@gmail.com>

* test(ecstore): synchronize batch cleanup metadata reads

Acquire object read locks while observing unversioned and explicit-version batch cleanup, then release them before waiting for progress. This prevents snapshots from spanning per-disk marker removal while retaining the existing quorum, timeout, and remote delete count assertions.

Validation: cargo fmt --all --check and git diff --check passed. The focused nextest test passed 20 stress iterations each with test-util and test-util,rio-v2, with retries disabled.

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>

* fix(scanner): wait for ABBA telemetry window (#7660)

Size the scanner collector wait budget from the requested sample count and interval instead of using a fixed 120 second timeout. This preserves the full duration/60+1 telemetry sample set for measured ABBA runs.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(ci): add fault-tolerance degradation suite to the functional chain

Adds a fault-tolerance suite that verifies read/write behavior under
drive and node loss against the erasure-coding contract and snapshots
health-endpoint responses at every degradation tier. Scenarios are
derived from product source (default_parity_count, erasure set sizing):

- A: single-node 4 drives (EC:2, read quorum 2): hide 1/2/3 drives
- B: multi-node 4x1 (one set of 4, EC:2): stop 1/2/3 nodes
- C: multi-node 4x4 (one set of 16, EC:4, read quorum 12): 1 node down
  lands exactly on the read-quorum boundary; 2 nodes down breaks it
- C2: multi-node 4x4 with RUSTFS_STORAGE_CLASS_STANDARD=EC:8 (read
  quorum 8, write quorum 9, lock majority 9): 2 nodes down puts reads
  inside the reported divergence window (read quorum met while the
  lock majority is broken)

By default a reads-refused-despite-met-read-quorum observation is
reported as known-divergence without failing the suite; the strict
input escalates it. Chain order becomes:
upgrade -> s3 -> kms -> tier -> storage -> heal -> pool -> security ->
replication -> fault-tolerance -> performance.

Covers the findings of the 2026-09 external degradation report
(reads at read quorum, health-endpoint readiness truthfulness,
degradedReasons capture) as automated regression probes.

* Revert "feat(ci): add fault-tolerance degradation suite to the functional chain" (#7666)

This reverts commit 6eb200022e.

* fix(ecstore): preserve multi-pool version histories (#7658)

* fix(ecstore): preserve multi-pool version histories

* fix(ecstore): satisfy clippy in rebalance test

* fix(ecstore): preserve decommission delete compatibility

* fix(heal): recover stale peer state after restart (#7665)

* fix(heal): select writable recovery intent owner (#7668)

* feat(ci): add fault-tolerance degradation suite to the functional chain (#7669)

* fix(get): reject pre-header read quorum failures (#7670)

* fix(get): reject pre-header read quorum failures

* fix(ci): unblock get pre-header quorum checks

* ci: stop defaulting suite dispatches to a pinned release (#7681)

The rustfs_version dispatch input defaulted to 1.0.0-rc.4-preview.1,
which shadowed the nightly fallback and started 404ing once that
release was deleted. Manual dispatches with no inputs now fall through
to the nightly package (same contract the pool suite already has);
passing rustfs_version or package_url still pins the build exactly as
before. Chain (repository_dispatch) runs are unaffected: the inputs
context is empty there, so they always used the nightly fallback.

* ci: manage backlog issues by signal instead of per-run filing (#7680)

* ci: manage backlog issues by signal instead of per-run filing

The suite workflows used to file one backlog issue per failed run
(dedup was by run ID, which never matched), so issues accumulated
without bound. Replace the inline filing step in every suite workflow
(s3, kms, tier, storage, heal, pool, security, replication, upgrade,
performance) with a single call to
auto-testing/scripts/issue_manager.py, which:

- dedups by signal: failing cases are searched among open issues by
  label (suite category + case ID); covered cases become a coalesced
  comment on the existing issue, only uncovered cases file a new one
- labels new issues with functional-test, the suite category, one
  label per failing case ID (lazily created), and env for
  bootstrap-class failures (no cases ran, wholesale failure, or
  404/ssh/clone/dpkg signatures in the log)
- closes open issues of the suite after a fully green run, citing the
  run as evidence; cancelled runs never file or close anything

The step is skipped cleanly when auto-testing (private checkout) does
not contain the manager, or when PF_TESTING_GH_TOKEN is unset.

* fix(ci): satisfy actionlint and workflow contract tests for the manager step

- heal and performance workflows have no rustfs_version dispatch input;
  referencing `${{ inputs.rustfs_version }}` in the manager step failed
  actionlint's expression type check. Their package source now resolves
  from package_url with the nightly fallback.
- scripts/test_security_workflow.py pinned the removed inline filing
  step. The wiring assertions now pin the manager step (manager path +
  per-suite report argument), and the evidence/stale-file tests assert
  the skip contract instead: without the private auto-testing checkout
  present, the step exits 0, publishes nothing, and leaves stale
  evidence untouched.

Verified locally: actionlint clean, shellcheck clean,
test_security_workflow.py 21/21.

* fix(storage): alias legacy meta bucket over internode rpc (#7685)

* fix(storage): alias legacy meta bucket over internode rpc

Retry read-only internode RPC metadata access from legacy .minio.sys to .rustfs.sys when mixed-version peers report missing metadata during rolling upgrades.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): target durable ILM receipt quorum fixture

Use the actual durable ILM receipt object path when taking target disks offline so the test exercises receipt write quorum instead of whichever set owns the source record path.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): stabilize cancelled rename test (#7684)

* test(scanner): report Linux descriptor drift ledger (#7692)

Add a read-only descriptor ledger mode to the Scanner/Heal Linux evidence planner so release operators can separate current-head measured descriptors from old-head measured artifacts and case-level inputs before final bundle assembly.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): retry recovery readback SlowDownRead (#7695)

Allow the Scanner/Heal interruption oracle to retry transient retryable GET failures after replacement recovery has converged. The readback still verifies exact object bytes and keeps a bounded timeout, so permanently unreadable objects continue to fail the case.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(pool): default the node-endpoints fallback to the 4-node fleet (#7696)

RUSTFS_POOL_NODE_ENDPOINTS has no secret/var configured, so the suite
ran with the workflow's inline 3-endpoint fallback and the explicit
--node-endpoints flag overrode the script default fixed in
rustfs/auto-testing#61 — every dispatch died at startup with 'must
provide at least 4 direct node endpoints' (run 34677538650). Add
rustfs-node4 to the fallback; explicit secret/var still wins.

* fix(scanner): require ABBA collector credentials (#7700)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): label G14 recovery read failures (#7701)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): preserve settings in token status (#7691)

* fix(heal): preserve settings in token status

* fix(heal): group status response context

* fix(heal): decode partial status settings

* ci: green runs for case failures; red only for harness breakdowns (#7698)

A failing product case used to turn the whole workflow red, so the run
conclusion carried no signal beyond 'something failed' and the report
was suppressed. New semantics across the functional suites:

- Suite steps run with continue-on-error: the outcome is still recorded
  for the report and the backlog issue manager (security/tier already
  carried the flag).
- Generate report always publishes the full per-case table plus a
  'Product result: N passed, M failed' summary, and its exit gate is
  harness health: red only when the suite never reached case level (no
  case verdicts), failed wholesale (zero passes, >=3 failures), or was
  cancelled/skipped. performance is unchanged (parked).
- tier's structured gate no longer fails on case failures; it keeps red
  for evidence-init and missing-gate-result breakdowns.
- pool/performance keep their existing red sources (install/benchmark).

Workflow contract tests updated to the new exit semantics: the security
report matrix keys green off the suite outcome, the evidence matrix
expects green for failure outcomes with recorded case rows (except
performance), the heal staged-rerun block expects the per-step table to
always publish, and run steps are now required to carry
continue-on-error.

Verified locally: actionlint clean; test_security_workflow.py 21/21.

* test(heal): retry G14 deferred outage put (#7703)

Treat SlowDownRead as a bounded retryable deferred PUT response after the target pool rejoins, and label terminal deferred outage PUT failures with stage context.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): retry G14 outage put probes (#7704)

Treat SlowDownRead as a retryable outage PUT probe response in the G14 multi-pool runner, and label terminal candidate failures with stage context.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): remove unused G14 put helper (#7705)

Remove the superseded ServiceUnavailable-only PUT helper after the G14 outage PUT probe switched to the shared retry classifier.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(tables): authorize reserved metadata reads (#7672)

* ci: harden report steps after first full new-semantics run (#7708)

First full serial pass with the green-on-case-failure semantics
(run 34693745171 / 34695021651) exposed three report-layer defects:

security: the report step referenced LOG_FILE, which is undefined in
this workflow (set -u killed the step before writing report.md), and
the verdict greps could not match the ANSI-escaped [PASS]/[FAIL] tags
in the real suite log. Point it at the artifacts suite.log, allow any
number of color escapes before the verdict tag, and count [SKIP]
lines separately (45 passed, 6 failed, 3 skipped was reported as an
unbound-variable crash).

pool: warp is stopped early (SIGINT) at the storage threshold and
only writes its final report on a clean exit, so an empty warp.log is
the expected shape of a healthy run - require its presence, not its
size. A mid-script die() abort or a FAIL step verdict must also turn
the validator red now that the run step is continue-on-error.

tier: add the standard 'Product result: N passed, M failed' summary
line computed from the case table, matching the other suites.

The contract test fixture previously injected LOG_FILE into the
environment and wrote verdict lines without ANSI escapes, which hid
both real-world defects; the fixture now mirrors the real suite
(stdout+tee with color tags) and asserts the pass/fail/skip counters.

* ci: tolerate ANSI reset escapes in security verdicts; fix tier product counts (#7712)

The overnight run after #7708 proved the security verdict greps still
counted zero: real verdict lines are '\e[1;31m[FAIL]\e[0m STS-105 ...'
— the reset escape sits between the tag and the case id, and the
pattern only tolerated escapes before the tag. Allow escapes on both
sides; the fixture now emits the reset too, mirroring the real suite.

The tier case table is produced by rustfs_tier_report.py and its rows
lead with the topology column, so the case-ID-first table grep matched
nothing ('Product result: 0 passed, 0 failed'). Parse the PASS/FAIL
counts from the '## Case Summary' bullets the report always emits,
falling back to a topology-aware table grep.

* fix(heal): reject overlapping admin heal owners (#7723)

* fix(heal): reject invalid admin selectors before admission (#7724)

* fix(heal): pass endpoints to overlap receipt test (#7738)

Reuse the heal-control endpoint fixture in the overlap receipt regression so the test matches the updated execution helper signature and selector validation boundary.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

Co-authored-by: hehutu <hehutu@gmail.com>

* fix(heal): verify inline shard bitrot during deep scans (#7731)

* fix(heal): verify inline shard bitrot during deep scans

* test(heal): fix CI lint and topology fixtures

* fix(heal): persist and retry partial-write repairs (#7729)

* fix(heal): persist and retry partial-write repairs

* test(heal): pass topology to overlap RPC tests

Use the existing coordinator endpoint fixture for the three overlap-test
calls to the endpoint-aware heal control executor. This repairs the E0061
test-build failure inherited from the release base.

* fix(scanner): prove dirty ACK after confirmed root write (#7732)

Allow a confirmed scanner root data-usage CAS write to prove its own publication when the follow-up root readback cannot provide a proof. Keep AlreadyDurable and all stale or companion paths on the existing readback-only proof boundary.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): rebuild truncated xl.meta from healthy quorum (#7730)

* fix(heal): rebuild truncated xl.meta from healthy quorum

* fix(test): pass topology to heal overlap RPC regression

* fix(test): drive heal admission alongside partial PUT

Poll the partial PUT and its mock heal receiver together, bound their handshake, and retain the existing repair-scope assertions.

* fix(test): prepare durable MRF fixtures and Linux heal stack

* fix(test): drive tier cleanup recovery after deferred attempts

---------

Co-authored-by: Hauser <housemecn@gmail.com>

* fix(heal): prove completed historical version cleanup (#7736)

* fix(heal): prove completed historical version cleanup

* test(heal): use debug runtime stack for C06 regression

* test(ecstore): acknowledge partial PUT heal admission concurrently

* test(mrf): fix journal fault and ownership fixtures

* fix(tables): retain dropped warehouse protection (#7675)

* fix(ecstore): gate shard integrity and preserve legacy recovery (#7741)

* fix(ecstore)!: bind bitrot shards to immutable part identities

Verify part, coding-index, and block identity across write, GET, and Heal
paths. Preserve identities across metadata-only copies and repair, include
them in multipart quorum selection, and require payload proof for receipts.

Keep legacy decoding with conservative parity and target-digest validation,
and document its unsupported cases and additional verification I/O.

BREAKING CHANGE: New bound-v1 shards require compatible readers throughout
the fleet. Legacy objects without sufficient integrity evidence return an
error; binary rollback after new writes requires verified data migration.

Refs: rustfs/backlog#2497

* fix(ecstore): preserve shard framing with independent integrity

Commit immutable part-generation Merkle roots and replicated proof indexes without changing existing checksum frames. Verify reads, reconstruction and Deep Heal against metadata quorum; keep legacy reads and explicitly defer unproven legacy data repair.

Preserve multipart rollback generations, require acknowledged durable index publication, and add decoder compatibility and donor-shard regression coverage.

* fix(heal): verify protected partial-write replay

* fix(heal): retire stale delete markers after bucket recreation (#7743)

* fix(heal): prove completed historical version cleanup

* test(heal): use debug runtime stack for C06 regression

* fix(heal): retire stale delete markers after bucket recreation

* test(ecstore): fix Clippy in retired marker regressions

* fix(heal): preserve historical null version identity (#7749)

* fix(s3): preserve version identities on suspended reads (#7751)

* fix(rebalance): retry contended runtime metadata access (#7752)

* fix(rebalance): retry contended runtime metadata access

* chore(rebalance): shrink typed error ratchet baseline

* fix(heal): bind bucket recovery to its original incarnation (#7744)

* fix(heal): bind bucket recovery to its original incarnation

* fix(heal): preserve stable bucket heal task errors

* fix(s3): classify copy source part read failures (#7746)

Map PartMissingOrCorrupt to SlowDownRead only at the GetObject/CopyObject source-reader boundary so quota metadata corruption keeps its internal fail-closed response.

Add store and e2e coverage for Harbor-style multipart staging CopyObject boundaries.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(tier): classify invalid remote credentials (#7753)

* fix(tier): classify invalid remote credentials

Classify deterministic remote-tier auth rejections before the probe cleanup fallback, and preserve the admin 4xx status for invalid credential responses.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): stabilize delete-marker movement regressions

Ignore target-local bucket incarnation fence metadata when comparing data-movement delete-marker identities, keep the release null-version listing fixture valid, and fix the nextest decommission-family filter so fault-hook tests actually join the serial group.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): serialize tier cleanup owner matrix (#7765)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): run bootstrap usage rebuild promptly (#7758)

Treat non-authoritative usage floor startup as pending bootstrap rebuild work so reset-published bootstrap markers cannot sit behind clean-idle or empty pause-backlog delay.

Wire recovery wakeups into the normal scanner cycle wait and expose the pending rebuild state in scanner status.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(tier): preserve cleanup scheduling for overwrites (#7766)

Queue committed tier free-version cleanup receipts for PUT and materialized CopyObject overwrites of transitioned null versions, while keeping remote deletion behind the existing persisted free-version cleanup path.

Tighten data-movement delete-marker retry equivalence by ignoring local bucket-incarnation fencing metadata, avoid retry fallback to the source pool, and keep version-list pagination from manufacturing an empty final page.

Refresh the e2e-distributed selector hash for the current release test set.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(release): backport main fixes and stabilize tier cleanup tests (#7793)

* fix(ecstore): stop pruning at nonempty directories (#7616)

* fix(ecstore): stop pruning at nonempty directories

* test(ecstore): release pruning fixtures before temp cleanup

(cherry picked from commit 8f150d1d8e)

* fix(heal): preserve retryable batch failures during recovery (#7642)

* fix(heal): preserve retryable batch failures during recovery

* test(heal): pin prebuilt hooks binaries in ci

(cherry picked from commit 5cd58319ed)

* fix(s3): reject oversize single PUT early and map body errors to 4xx (#7635)

* fix(s3): reject oversize single PUT early and map body errors to 4xx

A single PutObject above the 5 GiB single-request ceiling was only
rejected after the client had streamed 5 GiB into s3s's read-time body
budget, and the resulting BodySizeLimitExceeded surfaced from the erasure
writer as 500 InternalError. A body whose connection hit EOF before
Content-Length bytes arrived (hyper's IncompleteBody) was also a 500.
SDKs retry 500s, so one oversize upload was resent from offset 0 five
times.

- PutObject and UploadPart reject a declared length above
  MAX_SINGLE_PUT_OBJECT_SIZE with 400 EntityTooLarge before reading the
  body; the constant moves to rustfs_config so the s3s limit and the
  admission check share one value.
- ApiError maps BodySizeLimitExceeded to EntityTooLarge and a hyper body
  EOF to IncompleteBody across both io::Error conversions.

Fixes #7596.

* test(s3): cover UploadPart admission, aws-chunked length, real s3s limit

- Poll-counting test body proves PutObject and UploadPart reject a
  declared size above the ceiling with zero body polls; exact-cap and
  zero-length parts pass admission.
- A STREAMING-* aws-chunked PUT whose framed Content-Length exceeds the
  cap is admitted when the decoded length is within it and rejected when
  the decoded length is over it.
- The display-based BodySizeLimitExceeded matcher is checked against the
  real error produced by the pinned s3s Body budget.

(cherry picked from commit 50b31bc75b)

* fix(ecstore): make directory mtime fixture portable (#7623)

* fix(ecstore): make directory mtime fixture portable

* style(ecstore): format mtime fixture assertion

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
(cherry picked from commit b1cc286cac)

* fix(storage): prevent readiness after native migration failures (#7652)

* fix(storage): prevent readiness after native migration failures

* fix(storage): skip unsupported IAM records before reading

* fix(storage): use stable typed migration metadata errors

* fix(storage): include migration record in startup errors

* test(storage): cover native migration startup failures

* test(storage): use array chunks in migration fixture

---------

Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
(cherry picked from commit 0cbc3ffe61)

* fix(admin): expose OIDC account display fields (#7654)

Expose verified OIDC username and email claims as display-only metadata on self-account responses while preserving the virtual parent as the authorization identity.\n\nKeep rustfs-madmin public response structs unchanged by adding the optional wire fields through private handler response wrappers.

(cherry picked from commit f02bc947cd)

* fix(replication): correct peer joins and remote-state reporting (#7650)

* fix(replication): propagate verified peer deployment identities

* fix(replication): report actual remote peer state

* fix(replication): defer initial sync until all peers join

* test(replication): shut down TLS fixtures cleanly

---------

Co-authored-by: houseme <housemecn@gmail.com>
(cherry picked from commit 853ae63b6a)

* fix(s3): bound stalled UploadPart request bodies (#7659)

* fix(s3): bound stalled UploadPart request bodies

* fix(ci): preserve the S3S footprint ratchet

(cherry picked from commit 666dfd9f9f)

* fix(tables): reject reserved warehouse locations (#7671)

Co-authored-by: cxymds <cxymds@gmail.com>
(cherry picked from commit 414176c47f)

* fix(ci): bind nightly lanes to one resolved source (#7688)

(cherry picked from commit 01d8e4347f)

* fix(replication): close the pre-stable convergence gaps from backlog#2367 (#7626)

* fix(replication): total-order rule sort and honor V1 top-level Prefix

Rule matching had two defects from the pre-GA replication audit
(rustfs/backlog#2367 C-1 and C-2):

- The actionable-rule sort compared same-destination rules by priority but
  answered Equal for any other pair, which is not a total order; the
  standard library sort panics on such comparators once a slice exceeds the
  insertion-sort threshold, so an object matching more than 20 enabled rules
  across two or more targets could panic the PUT or DELETE task. Rules now
  sort by priority descending with destination and id as tie-breakers, and
  filter_target_arns preserves that order instead of draining a HashSet.

- A V1 rule written without a <Filter> carries its prefix at the top level;
  that field was never read, so <Prefix>logs/</Prefix> matched every object.
  ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping
  precedence. The existing prefix fixtures were built this way and had been
  asserting nothing.

* fix(admin): advertise data-usage and listen capabilities to rc

The rc client gated `rc du` and `rc watch` on a pinned contract that
matched server versions by the string prefix `1.0.0-rc.`; a server that
reports `1.0.0` no longer matches, and the dynamic `advertised` list did
not carry either name, so `rc du` against a GA server fails with an
unsupported-capability error (rustfs/backlog#2367 E-2).

Advertise `admin.data-usage` from the admin route inventory like the IAM
entries, and `listen_notification` for the bucket `?events=` extension
route the admin router dispatches. The client merges advertised entries
ahead of its pinned contract, so no version sniffing is needed.

* fix(site-replication): stop notifying the local site on remove and rotate

The pending-remove and pending-rotation notification loops skipped the
local site by endpoint only, while finalization identifies it by
deployment id or endpoint. The reconcile tick resolves the local peer from
the node's own listen address (and a handler from the request Host), so
`remove --all` dialed the site's registered endpoint, waited out the
request timeout against the lifecycle lock it was holding, and answered
`Partial: failed to notify 1 peer(s)` for a removal that had succeeded
(rustfs/backlog#2367 A-4, backlog#2195 item 3).

Both loops now iterate the peers still awaiting notification through one
helper that applies the finalization identity.

* fix(site-replication): promote and settle IAM retries without a tick of slack

Two retry-queue behaviours kept an IAM change from converging for ten to
twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3,
backlog#2305):

- The lightweight 30-second pass filtered its reachability probe to bucket
  ops, so a backed-off IAM or bucket-metadata snapshot waited for the
  600-second tick to notice the peer. It now probes every backed-off class
  and still replays only bounded bucket ops; promotion is a state flip the
  heavyweight tick acts on.

- Backoffs are multiples of the tick interval, so a failure stamped δ
  seconds after a tick was 600 − δ old at the next tick and slipped a whole
  extra interval. The heavyweight drain now evaluates backoff halfway to its
  next tick.

- An IAM entry first created by a non-deletion failure (the add bootstrap's
  snapshot send, the drain's own replay, an import-iam schedule) was never
  stamped `deletions_recorded`, so a later recorded deletion could not
  settle it and it escalated to the marker only `replicate repair` clears.
  Entries created by this binary now start recorded; a row persisted by an
  older binary keeps the escalation semantics.

* fix(site-replication): reload peer node caches after bucket wiring writes

Every S3 bucket-config write ends by asking the other nodes of the cluster
to reload the bucket's metadata; the site-replication writers never did.
On a multi-node site the node that ran the pairing (or applied a peer's
bucket-meta item) rewrote the bucket targets and the derived replication
rules on disk, while every other node kept serving its cached copy for up
to the 15-minute refresh. A `resync start` routed to such a node reported
every freshly wired bucket as `Config not found` and a bucket whose
operator target the pairing had replaced as `recorded remote target no
longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional
SITE-105).

Add one best-effort reload helper in the site-replication hooks and call it
after the bucket setup, versioning, peer bucket-meta apply, removed-peer
cleanup, make-with-versioning, and endpoint-refresh writes; the ensure
helpers now report whether they wrote so unchanged passes stay silent. The
resync manifest and start now read the persisted wiring instead of the
node-local cache, matching the target read the start path already did.

The new four-node e2e pairs two clusters and starts a resync through a
non-coordinator node right after pairing; it also covers an IAM user
created on a non-coordinator node converging to the peer site.

* test(e2e): cover delete-marker replication from a multi-node source

The functional suite reported delete markers created on a 3-node source
never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report
was a probe defect, but the shape had no coverage: the existing
delete-marker e2e runs a single-node source. Pin it against a four-node
source replicating to a four-node peer and to a single-node target, with
the write and the delete issued through different nodes.

* ci(e2e): refresh the distributed selection for the new replication cases

Four distributed cases were added (two site-replication, two delete-marker
replication). The linux digest is derived from the last CI listing of the
lane (34 cases, matching the previous pin) plus the four new names; the
darwin digest is the local listing, which selects the same 38 cases.

(cherry picked from commit aeaba86d73)

* fix(e2e): require a verified server binary for every e2e run (#7687)

* fix(ci): share quick checks and lint workflows

* fix(ci): install actionlint from its verified release

* fix(ci): reject dependencies on required quick checks

* feat(test): verify the E2E server build and source identity

* test(e2e): register verified Darwin test membership

* test(e2e): record verified Linux receipt test membership

* test(e2e): record compiled Darwin receipt test membership

* test(e2e): record compiled Linux receipt test membership

* test(e2e): record Darwin e2e-full membership after merging main

* fix(test): route scanner/heal evidence E2E runs through the verified server binary

The evidence runners built rustfs with plain cargo and then ran e2e_test directly, which now fails without a run receipt. They build through scripts/e2e_binary.py and run the e2e_test invocations under e2e_binary.py run; the obsolete rustfs.features stamp is removed.

* docs(e2e): run server-backed e2e commands through the verified binary wrapper

* test(e2e): record Linux e2e-full membership from the branch CI listing

(cherry picked from commit 2909b1bfe1)

* fix(kms): classify KMS/SSE error contracts and SSE-S3 headers (#7697)

* fix(sse): classify bare SSE-KMS writes when no KMS is available

A `aws:kms` request without a key id, on a bucket without a default key,
returned `500 InternalError` whenever no KMS service was running: the
"no KMS key available" branch exited with an untyped storage error before
the availability classification that the keyed form already received.

Route that branch through the same split: `503 ServiceUnavailable` while
a configured KMS is stopped, `400 InvalidRequest` when KMS was never
configured, and `400 InvalidRequest` naming the missing key id when a
running KMS has no default key. `CreateMultipartUpload` shares the path.

Adds a unit test for the bare form and an e2e module that stops KMS
through the admin API, runs a master-key-only node, and runs a Local KMS
without a default key; refreshes the e2e-full selection digests.

(cherry picked from commit c3259dadc3d603a9185a5b0ad9f83dfb884e61c8)

* fix(sse): keep KMS error classes on the encrypted read path

GetObject, CopyObject and UploadPartCopy on an SSE-KMS object whose key
no longer exists answered `500 InternalError` ("KMS key not found") while
PutObject under the same key already answered `400 KMS.NotFoundException`.
The read path carries its classification through ecstore's
`EncryptionResolutionErrorKind`, which had no kind for a missing key, a
denied KMS grant or a missing backend capability, so all three folded
onto `DecryptionFailed` and the S3 layer reported an internal fault.

Add `KeyNotFound`, `AccessDenied` and `NotImplemented` kinds, map them on
both sides of the boundary, and give an envelope the configured backend
cannot unwrap a diagnosable message while keeping its `500`.

Unit tests cover the kind round trip and the reader wrapping; a new e2e
test deletes a key immediately and checks GET/Copy return 400 with
`KMS.NotFoundException` while HEAD stays 200. The e2e-full selection
digests are refreshed from the current listing (the previous digests
predated the delete-authorization tests) and the e2e `create_default_key`
helper is updated to the accepted `EncryptDecrypt` spelling.

(cherry picked from commit 2523a9814e97caea318d4ff1a51bef3a4d4445b2)

* fix(kms): classify key-management errors on the admin routes

`POST /kms/keys`, the legacy `create-key` alias and `generate-data-key`
reported every backend refusal as `500`: a blank key name (which each
backend failed on differently, the Local backend by writing a key file
with an empty stem), a name already taken, an unknown key, a disabled key
and a capability the backend lacks. `delete` and the lifecycle routes
already classified the same errors.

Refuse a blank or whitespace name in `KmsManager::create_key` before any
backend sees it, and share one `KmsError` to status mapping across
create, delete and generate-data-key (400 for validation and key state,
404 for an unknown key, 409 for a taken name, 501 for a missing
capability, 500 only for damaged material). The XML-error routes carry
the same status explicitly since s3s derives none for a custom code.

The read-only Static backend now reports create, delete and
cancel-deletion as `UnsupportedCapability`, matching its rotate and
enable/disable answers, so the admin API returns 501 for all of them.

(cherry picked from commit e33cac5493c4d9d6662e0d2980b58ba2b24a6d1b)

* fix(sse): stop SSE-S3 responses from naming the wrapping KMS key

`x-amz-server-side-encryption-aws-kms-key-id` is defined for `aws:kms`
objects only, but PutObject, CopyObject, CreateMultipartUpload and
GetObject returned it for `AES256` objects too, carrying the KMS key that
wraps the SSE-S3 data key (the service default, or the literal `default`
on a node without KMS). The write paths copied `kms_key_id` from the
encryption material unconditionally, and the single-decrypt GET
classification did the same after resolving the key for authorization.

Add `EncryptionMaterial::response_kms_key_id`, which yields the id only
for SSE-KMS, use it at the four write-response sites, and gate the GET
classification the same way. CompleteMultipartUpload and HeadObject
already omitted the header.

Unit tests pin both directions; a new e2e test covers Put/Get/Head/Copy
and CreateMultipartUpload for AES256 with an aws:kms control. The
e2e-full selection digests are refreshed from the current listing.

(cherry picked from commit 29d793a63352b0b60fd53c565e80fdbede8964bb)

* fix(s3): validate PutBucketEncryption rules before storing them

A default-encryption rule naming an unknown `SSEAlgorithm` (for example
`AES128`), a rule without `ApplyServerSideEncryptionByDefault`, an empty
rule list, or a `KMSMasterKeyID` on an `AES256` rule was stored as
written: the only algorithm check on the route decided whether to fill
in the default KMS key. `GetBucketEncryption` then advertised that
configuration while the write path encrypted header-less writes under
its `AES256` fallback, so the bucket's declared and actual schemes
disagreed. Two comments claimed the route already refused unknown
algorithms.

Validate the configuration before any of it is applied: `MalformedXML`
for a malformed rule set or unknown algorithm, `InvalidArgument` for a
key id on a non-KMS rule, and nothing stored on refusal. Correct the two
comments to describe when the AES256 fallback is still reachable.

Unit tests cover every refusal and the accepted shapes; an e2e test
checks the refusals leave the previous configuration in place. The
e2e-full selection digests are refreshed from the current listing.

(cherry picked from commit 29e4486dce41197ed93f5253cdbabc57d27a4ddb)

* test(e2e): refresh e2e-full selection for the combined KMS/SSE fixes

* test: align two unit tests with the new KMS and bucket-encryption contracts

`scheduled_deletion_carries_a_deadline_and_can_be_cancelled` still
expects the state error (`InvalidOperation`) for cancelling a key that
is not pending deletion; only the Static backend's mutations moved to
`UnsupportedCapability`. The uninitialized-store PutBucketEncryption
test now sends a well-formed AES256 rule so it reaches the store lookup
instead of the new configuration validation.

(cherry picked from commit e2e6a2535a)

* fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over (#7709)

* fix(site-replication): keep an operator's bucket-level target to a peer instead of taking it over

Site replication wired each bucket by looking for an existing replication
target "to the same peer" and rewriting the first match in place as its own
same-name target. An operator's bucket-level target that happened to point
at that site (different target bucket, operator credentials) was the first
match whenever it pre-dated the join, and the reconciler repeats the pass
every 600s, so the takeover also depended on target order afterwards. The
operator's rule then named an ARN no target backed and their bucket
replication stopped silently, while the inherited bucket-level reset id
made every site resync report the bucket as owned by another resync
(rustfs/backlog#2479, rustfs/backlog#2489).

Follow MinIO's `getRemoteARN` / `getRemoteARNForPeer` shape instead:

- Wiring updates a target in place only under the same ARN, or when it is
  recognisably the site's own under an older ARN shape (same peer,
  same-name target bucket, site replication service account). Anything
  else gets the site target added next to it.
- The site resync manifest takes the target the derived
  `site-repl-<deployment id>` rule names (same-name shape as fallback), so
  an operator target to the peer neither aborts the bucket as "multiple
  remote targets matched peer" nor gets resynced into.
- Peer removal prunes only targets a pruned derived rule names or the
  same-name target bucket; operator targets stamped with the peer's
  deployment id survive together with their rules.

Unit tests cover the three predicates. e2e
`test_site_replication_keeps_operator_bucket_target_to_peer` runs a
bucket-level replication plus `replication-reset` to the future peer, joins
the sites, and requires the operator target untouched, both paths
delivering, the site resync completing against the site target, and the
operator target and rule surviving `replicate remove --all`; without the
fix it fails at the join with the operator target gone. The repl-nightly
selection digest is refreshed for the new case.

* test(site-replication): drop a redundant clone flagged by clippy

The reconcile unit test cloned the remote peer into the state map although
the binding is not used afterwards; workspace clippy (-D warnings) rejects
that as redundant_clone.

(cherry picked from commit ecdc55fa4b)

* fix: enforce S3 permissions for recursive force deletion (#7661)

* fix: enforce S3 authorization for recursive deletion

* fix: satisfy the s3s footprint guard

* fix: restore list versions policy compatibility (#7686)

(cherry picked from commit 3fd1ce414d)

* fix(ci): repair functional defaults and chain regression checks (#7664)

* fix(ci): default functional suites to nightly packages

* test(ci): follow the fault-tolerance chain handoff

(cherry picked from commit 509a0fa90c)

* fix(ci): align security workflow tests with chain (#7679)

(cherry picked from commit d9e47d2813)

* test(ecstore): keep tier cleanup tests stable after immediate receipt queueing

Release PR #7766 made PUT/CopyObject overwrites queue the tier free-version cleanup receipt immediately, which broke two ecstore tests on release CI. In tier_overwrite_put_and_self_copy_recover_persisted_cleanup_owners the restarted store already runs expiry workers from the second iteration on, so they deleted the remote bytes before the test could assert that the commit leaves them in place; the test now fails the first remote DELETE via set_remove_failure(true) so the cleanup owner stays durable and the later restart still has to rediscover it from xl.meta (the failed remove does not bump remove_count, and the test re-enables removes before the recovery wait). In batch_transitioned_delete_post_commit_failures_roll_back_without_free_version_receipt the convergence loop now treats a transient InsufficientReadQuorum as "not yet converged", because cleanup rewrites xl.meta disk by disk and a racing read can briefly miss quorum (seen in the rio-v2 lane); any other error still panics.

* Revert "fix(ci): align security workflow tests with chain (#7679)"

This reverts commit 4544359f6d.

* Revert "fix(ci): repair functional defaults and chain regression checks (#7664)"

This reverts commit 5ba7ec0291.

* fix(ecstore): pass shard integrity to backported ingest-mode test

The stalled-reader test backported with #7659 used main's four-argument encode_with_ingest_mode, but release's signature takes an optional IntegrityBuilder, so pass None to keep the test focused on ingest-mode cleanup.

---------

Co-authored-by: Henry Guo <marshawcoco@gmail.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: RJ Regenold <rregenold@teamraft.com>
Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: Jason Kossis <jkossis@gmail.com>

* fix(ci): restore release test compilation and e2e membership (#7799)

* fix(s3): return stored content language on local GET (#7755)

* fix(s3): return resolved object tagging version identities (#7757)

* fix(tables): bound warehouse index recovery (#7677)

Co-authored-by: Chris <anzhengchao@gmail.com>

* fix(upgrade): probe legacy IAM namespace before migration (#7816)

* fix(release): backport IAM migration startup fixes and recent main fixes (#7825)

* fix(auth): reject unsigned x-amz headers on header-signed SigV4 requests (release) (#7813)

* fix(s3): preserve missing version errors on object reads (#7800)

* fix(s3): preserve missing version errors on object reads

* test(s3): align missing-version assertions with read contract

* fix(heal): bind admin traversal to bucket incarnation (#7802)

* fix(heal): bind admin traversal to bucket incarnation

* fix(ecstore): reopen healthy hedged readers after peer loss

* test(ecstore): isolate tier overwrite recovery scheduling

* test(ecstore): observe tier cleanup under object read locks

* fix(ecstore): bound decommission entry tracing

* test(ecstore): drain incarnation heal fixture writes

* test(heal): match debug server stack in deep heal fixtures

* test(heal): include identities in C06 count failures

* test(heal): drain PUT tails before inspecting B920 fixtures

* test(scanner): isolate retained MRF retry slots

* fix(ecstore): separate PUT cleanup intent from persisted metadata

* fix(ecstore): capture failed object lock acquisitions (#7803)

* fix(ecstore): capture failed object lock acquisitions

* test(heal): include identities in C06 count failures

* test(heal): drain PUT tails before inspecting B920 fixtures

* fix(ecstore): reopen healthy hedged readers after peer loss

* test(ecstore): isolate tier overwrite recovery scheduling

* test(ecstore): observe tier cleanup under object read locks

* fix(ecstore): bound decommission entry tracing

* test(ecstore): drain incarnation heal fixture writes

* test(heal): match debug server stack in deep heal fixtures

* test(scanner): isolate retained MRF retry slots

* fix(ecstore): separate PUT cleanup intent from persisted metadata

* fix(heal): retain terminal outcomes across restart (#7805)

* fix(heal): retain terminal outcomes across restart

* test(heal): use octal mode for terminal recovery fixture

* fix(ecstore): reopen healthy hedged readers after peer loss

* test(ecstore): isolate tier overwrite recovery scheduling

* test(ecstore): observe tier cleanup under object read locks

* fix(ecstore): bound decommission entry tracing

* test(ecstore): drain incarnation heal fixture writes

* test(heal): match debug server stack in deep heal fixtures

* test(heal): include identities in C06 count failures

* test(heal): drain PUT tails before inspecting B920 fixtures

* test(scanner): isolate retained MRF retry slots

* fix(ecstore): separate PUT cleanup intent from persisted metadata

* fix(heal): recover replacement ownership across VM restarts (#7810)

* fix(heal): recover replacement ownership across VM restarts

* test(ecstore): isolate tier overwrite recovery scheduling

* test(ecstore): observe tier cleanup under object read locks

* fix(ecstore): bound decommission entry tracing

* test(ecstore): drain incarnation heal fixture writes

* fix(ecstore): reopen healthy hedged readers after peer loss

* test(heal): match debug server stack in deep heal fixtures

* test(heal): include identities in C06 count failures

* test(heal): drain PUT tails before inspecting B920 fixtures

* test(scanner): isolate retained MRF retry slots

* fix(ecstore): separate PUT cleanup intent from persisted metadata

* fix(tables): isolate protected S3 listings (#7676)

Co-authored-by: overtrue <anzhengchao@gmail.com>

* fix(ci): restore Linux E2E selection and split offline enrollment (#7843)

* fix(heal): assign stable identities to scanner repair requests (#7823)

* fix(heal): assign stable identities to scanner repair requests

* test(s3): isolate UploadPart inactivity timing

* fix(heal): backport pool metadata scope to release (#7846)

* fix(heal): scope replacement pool metadata to its owning set

(cherry picked from commit a72a75569c)

* fix(heal): use domain imports in pool metadata regression test

(cherry picked from commit e8c5448e21)

* test(heal): set pool metadata regression recursion limit

---------

Co-authored-by: Hiroaki KAWAI <1468181+hkwi@users.noreply.github.com>

* fix(heal): persist pool/set outcomes and recovery scope (#7839)

* fix(heal): persist pool/set outcomes and recovery scope

* test(heal): verify selector receipts and process-crash recovery

* fix(heal): scope replacement pool metadata to its owning set

(cherry picked from commit a72a75569c)

* fix(heal): use domain imports in pool metadata regression test

(cherry picked from commit e8c5448e21)

* test(heal): set pool metadata regression recursion limit

---------

Co-authored-by: Hiroaki KAWAI <1468181+hkwi@users.noreply.github.com>

* fix(ecstore): scope recursive walks to object prefixes (#7850)

* fix(heal): scope nested walks and reject admin read-repair (#7851)

* fix(heal): scope nested walks and reject admin read-repair

* fix(ci): allocate stack for heal metadata regression

* test: cover overlapped multipart staging copy (#7865)

Add Harbor-style E2E coverage for a CopyObject racing with CompleteMultipartUpload. The test allows the copy to observe an unpublished source as NoSuchKey, but rejects 5xx leakage and proves retrying the completed source remains readable.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(release): repair acceptance checks and prepare 1.0.0 (#7866)

* test(e2e): stabilize release acceptance checkpoints

* chore(release): prepare 1.0.0

* test(scanner): stabilize W13 release evidence gates (#7870)

Add the object-only completion diagnostic case and make the evidence runner resolve CARGO_TARGET_DIR before recording the rustfs binary path.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(kms): cover non-default Vault Transit paths (#7657)

Backport the Vault coverage from 4d31621f0a. Keep the release cleanup polls synchronized with object read locks.

* fix(replication): count a bodiless 405 as a replicated delete marker (#7756)

* fix(replication): accept a bodiless 405 as a replicated delete marker during resync

A resync verifies each delete marker with `HEAD ?versionId=<marker>` on the
target. S3 targets (RustFS, MinIO, AWS) answer that with 405 and no body,
and the SDK only synthesizes an error code for 404, so the error arrived
with `code() == None`. `is_retryable_delete_replication_head_error` then
treated it as an ambiguous failure: every delete marker counted as a failed
object with `target service error`, and a site resync over any bucket that
holds a delete marker reported the whole bucket as failed
(rustfs/backlog#2479, SITE-105 on rc.6 and nightly).

Use the raw HTTP status the way the 404 path already does: a 405 without a
code confirms the marker propagated. Ambiguous statuses still fail.

- Unit tests drive `verify_resync_head_result` against a scripted target
  answering 405 (accepted) and 503 (still failed).
- e2e `test_site_replication_resync_replicates_delete_marker` joins two
  sites, converges a live object and a delete marker, and requires the site
  resync to complete with zero failed objects; the repl-nightly selection
  digest is refreshed for the new case.

* fix(replication): verify marker-version purges by absence during resync

A delete-marker resync entry with a pending or failed version purge asks
the target to remove the marker, so the bodiless 405 that proves a
created marker propagated proves the purge did not happen. Accept the
405-as-success mapping only for marker creation (empty
version_purge_status); a purge counts as replicated only when the target
answers not found, and any other HEAD outcome stays failed. Unit tests
drive both purge statuses against the 405 fixture and the 404 fixture.

(cherry picked from commit 1ef3974bad)

* fix(kms): refuse key ids that leave the key prefix on the Vault backends (#7727)

Only the Local backend refused a key identifier containing a path
separator. On Vault KV2 a create with the name bad/name succeeded and
produced a nested KV2 path that the listing then reported as a
directory rather than a key, and an identifier containing .. addressed
a record outside the configured key prefix once the HTTP client
normalised the URL; Vault Transit built its transit key name and its
metadata path from the same unchecked identifier.

Lift the Local backend's containment rule into a shared segment check
(empty, /, backslash, NUL, . and ..) and apply it at the single point
where each backend turns the identifier into a path or a Transit key
name, so create, describe, encrypt, delete and the metadata writes all
refuse with InvalidKey before any request reaches Vault. The admin API
already maps that to 400. The AWS backend is untouched: it addresses
keys by ARN and alias, both of which contain /.

Refs rustfs/backlog#2474 (KMS-213 CreateNegatives on vault-kv2).

Co-authored-by: Hauser <housemecn@gmail.com>
(cherry picked from commit 2f2e775655)

* fix: resume GET body after peer short EOF (#7852)

Backport GET recovery from afc8861fec. The Connect TLS classification change requires the main-only proxy transport layer.

* fix(s3): honor presigned UploadPart checksum queries (#7748)

* fix(s3): honor presigned UploadPart checksum queries

* fix(s3): keep checksum errors within gateway boundary

---------

Co-authored-by: Hauser <housemecn@gmail.com>
(cherry picked from commit a8ace89516)

* fix(table-catalog): map transient lock failures to unavailable (#7841)

(cherry picked from commit 5c73a3d602)

* fix(ecstore): preserve ready read errors during encoding (#7798)

Backport the erasure fix from c95b65a7fc while retaining release shard integrity support. Dependency upgrades and the main-only Connect proxy and diagnostic fixes are excluded.

* ci: add the S3 Tables (Iceberg REST Catalog) functional suite workflow (#7873)

New workflow rustfs-table-test.yml following the shared suite contract:
Initialize functional evidence (FUNCTIONAL_ARTIFACTS_DIR/LOG_FILE/
REPORT_FILE/TMPDIR), checkout for the report parser, auto-testing
clone with retry, continue-on-error on the run step, report generation
through scripts/functional_case_report.py with the product gate (case
failures stay green; harness breakdowns turn red), dashboard upload
under functional-reports/table/, backlog issue manager wiring
(--suite table --category table), artifact upload (report.md, cases.md,
suite.log), before/after environment cleanup, and the chain handoff
step (repository_dispatch only, next: pool expansion).

The suite script itself lives in rustfs/auto-testing (TBL-101..107,
paired PR). The run step passes the product smoke script path
(scripts/table-catalog/pyiceberg_smoke.py) via --smoke-script and the
log via --log-file, matching the other suites.

Contract tests: the table suite is registered in the generic evidence
and workflow matrices (JOBS/DIRECT_TESTS/allowlist) plus a dedicated
TableSuiteTests wiring check. Full run: 21/21.

* test(ecstore): tolerate slow read-repair diagnostics CI (#7892)

* fix(heal): report verified normal shard repairs (#7890)

* fix(heal): retry interrupted internode shard writes

(cherry picked from commit 0bd75f47b6326edd8167ee43b8776f6276692e92)

* fix(deps): update rustls for RUSTSEC-2026-0285

(cherry picked from commit 938e1259d13da7832d85ac5c066a59ab3a28bc63)

* test(heal): wait for replacement disk and coordinator readiness

(cherry picked from commit feeaa6f053aac4b7c8e32477c0ab5ebbd3dcdff3)

* fix(heal): retain RPC transport failures across metadata probes

(cherry picked from commit f48ec9f0c6e37b1f941620c666a9357c4b892d68)

* fix(s3): resume GET bodies after erasure read quorum loss

(cherry picked from commit a38e859905)

* test(heal): keep outage objects in the replacement erasure set

(cherry picked from commit a669f50a366e35505e7745f8b680d8ecf1aa31be)

* fix(rpc): retry authenticated requests after peer epoch changes

Backport the stale-epoch recovery fix from main follow-up #7889. Keep invalid signatures and nonce replays terminal.

* test(rpc): assert retryable rejection of stale boot epochs

(cherry picked from commit 1c4b39b99)

* ci: bound concurrent heal clusters in the full E2E gate

Apply the #7889 scheduling fix while preserving release test membership and zero retries.

* fix(ecstore): hide pending purge entries from listings (#7895)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): scope pending purge listing regression (#7897)

Keep the store-level pending purge listing regression focused on the ordinary recursive path and exact-prefix shortcut. The deeper delimiter CommonPrefix case is a separate physical-walker boundary and should not leave the release test suite red.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(obs): expose scanner usage freshness metrics (#7896)

Add Prometheus scanner metrics for dirty-usage pending buckets, usage-cache save/publication timestamps, publication result codes, and deferred publication state. This gives operators a direct scrape path for confirming scanner dirty ACK behavior without relying on admin JSON.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): backport delete quorum rollback to release (#7902)

* ci: rotate the shared functional-tests concurrency group to -v2 (#7900)

* fix(s3): render null write version IDs consistently (#7904)

* fix(heal): quarantine corrupt root recovery records (#7905)

* fix(heal): quarantine corrupt root recovery records

Keep invalid root-heal records durable while isolating them from healthy recovery owners. Serialize startup replay with administrator admission and fence quarantined task IDs so independent compensation heals can proceed safely.

* fix(ci): align release workflow contract tests

* fix: satisfy root recovery clippy checks

* fix(heal): fence canonical repair receipts (#7909)

* fix(heal): fence canonical repair receipts

* fix(heal): preserve unversioned receipt identity

* fix(heal): persist replacement intent on remote survivor (#7912)

* chore(release): merge main into release (#7913)

* feat(nightly): publish packages as assets of the rolling 'nightly' release (sync from release) (#7593)

feat(nightly): publish packages as assets of the rolling 'nightly' release (#7592)

Replace the assets-branch scheme with a proper GitHub Release on
rustfs/auto-testing: a single 'nightly' release whose deb/rpm assets
are replaced in place on every build. This is the standard channel —
visible on the repo's Releases page, stable download URLs, no git
history growth (release assets live outside the repository).

- New scripts/release/publish_nightly_assets.sh: resolves-or-creates
  the 'nightly' release via the REST API, deletes same-name assets,
  uploads rustfs-nightly-latest.{deb,rpm}, then PATCHes the release
  body with the build provenance (ref@sha, run link, sizes, SHA256).
  Plain curl + python3, no gh CLI (the build fleet has none — #7586).
- The workflow step shrinks to invoking the script; full flow
  exercised end-to-end against the real release with probe files
  (create / upload / overwrite / download round-trip / body update).

* fix(ecstore): stop pruning at nonempty directories (#7616)

* fix(ecstore): stop pruning at nonempty directories

* test(ecstore): release pruning fixtures before temp cleanup

* fix(replication): close the pre-stable convergence gaps from backlog#2367 (#7626)

* fix(replication): total-order rule sort and honor V1 top-level Prefix

Rule matching had two defects from the pre-GA replication audit
(rustfs/backlog#2367 C-1 and C-2):

- The actionable-rule sort compared same-destination rules by priority but
  answered Equal for any other pair, which is not a total order; the
  standard library sort panics on such comparators once a slice exceeds the
  insertion-sort threshold, so an object matching more than 20 enabled rules
  across two or more targets could panic the PUT or DELETE task. Rules now
  sort by priority descending with destination and id as tie-breakers, and
  filter_target_arns preserves that order instead of draining a HashSet.

- A V1 rule written without a <Filter> carries its prefix at the top level;
  that field was never read, so <Prefix>logs/</Prefix> matched every object.
  ReplicationRuleExt::prefix now falls back to it, with a <Filter> keeping
  precedence. The existing prefix fixtures were built this way and had been
  asserting nothing.

* fix(admin): advertise data-usage and listen capabilities to rc

The rc client gated `rc du` and `rc watch` on a pinned contract that
matched server versions by the string prefix `1.0.0-rc.`; a server that
reports `1.0.0` no longer matches, and the dynamic `advertised` list did
not carry either name, so `rc du` against a GA server fails with an
unsupported-capability error (rustfs/backlog#2367 E-2).

Advertise `admin.data-usage` from the admin route inventory like the IAM
entries, and `listen_notification` for the bucket `?events=` extension
route the admin router dispatches. The client merges advertised entries
ahead of its pinned contract, so no version sniffing is needed.

* fix(site-replication): stop notifying the local site on remove and rotate

The pending-remove and pending-rotation notification loops skipped the
local site by endpoint only, while finalization identifies it by
deployment id or endpoint. The reconcile tick resolves the local peer from
the node's own listen address (and a handler from the request Host), so
`remove --all` dialed the site's registered endpoint, waited out the
request timeout against the lifecycle lock it was holding, and answered
`Partial: failed to notify 1 peer(s)` for a removal that had succeeded
(rustfs/backlog#2367 A-4, backlog#2195 item 3).

Both loops now iterate the peers still awaiting notification through one
helper that applies the finalization identity.

* fix(site-replication): promote and settle IAM retries without a tick of slack

Two retry-queue behaviours kept an IAM change from converging for ten to
twenty minutes after a peer came back (rustfs/backlog#2367 A-1 and A-3,
backlog#2305):

- The lightweight 30-second pass filtered its reachability probe to bucket
  ops, so a backed-off IAM or bucket-metadata snapshot waited for the
  600-second tick to notice the peer. It now probes every backed-off class
  and still replays only bounded bucket ops; promotion is a state flip the
  heavyweight tick acts on.

- Backoffs are multiples of the tick interval, so a failure stamped δ
  seconds after a tick was 600 − δ old at the next tick and slipped a whole
  extra interval. The heavyweight drain now evaluates backoff halfway to its
  next tick.

- An IAM entry first created by a non-deletion failure (the add bootstrap's
  snapshot send, the drain's own replay, an import-iam schedule) was never
  stamped `deletions_recorded`, so a later recorded deletion could not
  settle it and it escalated to the marker only `replicate repair` clears.
  Entries created by this binary now start recorded; a row persisted by an
  older binary keeps the escalation semantics.

* fix(site-replication): reload peer node caches after bucket wiring writes

Every S3 bucket-config write ends by asking the other nodes of the cluster
to reload the bucket's metadata; the site-replication writers never did.
On a multi-node site the node that ran the pairing (or applied a peer's
bucket-meta item) rewrote the bucket targets and the derived replication
rules on disk, while every other node kept serving its cached copy for up
to the 15-minute refresh. A `resync start` routed to such a node reported
every freshly wired bucket as `Config not found` and a bucket whose
operator target the pairing had replaced as `recorded remote target no
longer exists` (rustfs/backlog#2367 A-5, backlog#2195 item 2; functional
SITE-105).

Add one best-effort reload helper in the site-replication hooks and call it
after the bucket setup, versioning, peer bucket-meta apply, removed-peer
cleanup, make-with-versioning, and endpoint-refresh writes; the ensure
helpers now report whether they wrote so unchanged passes stay silent. The
resync manifest and start now read the persisted wiring instead of the
node-local cache, matching the target read the start path already did.

The new four-node e2e pairs two clusters and starts a resync through a
non-coordinator node right after pairing; it also covers an IAM user
created on a non-coordinator node converging to the peer site.

* test(e2e): cover delete-marker replication from a multi-node source

The functional suite reported delete markers created on a 3-node source
never reaching the target (rustfs/backlog#2195 item 4, REP-105). The report
was a probe defect, but the shape had no coverage: the existing
delete-marker e2e runs a single-node source. Pin it against a four-node
source replicating to a four-node peer and to a single-node target, with
the write and the delete issued through different nodes.

* ci(e2e): refresh the distributed selection for the new replication cases

Four distributed cases were added (two site-replication, two delete-marker
replication). The linux digest is derived from the last CI listing of the
lane (34 cases, matching the previous pin) plus the four new names; the
darwin digest is the local listing, which selects the same 38 cases.

* feat(console): support a configurable console URL prefix (#7634)

Co-authored-by: houseme <housemecn@gmail.com>

* chore(release): merge release into main for rc.6 (#7638)

* test(heal): cover MRF manifest CAS legacy transition (#7443)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ci): enable release branch checks and repair test imports (#7441)

* test(scanner): structure heal release evidence lanes (#7442)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): require segment producer identities (#7444)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): expose recovery intent identity (#7445)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): summarize failed heal perf reports (#7446)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(common): retain unmatched MRF repair proofs (#7447)

Add a recorded verified-repair consumer that discharges only exact retained anchors for the requested bucket while leaving unmatched proofs in the event ring. This gives the future durable successor writer a fail-closed primitive before any tombstone or GC path is enabled.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ci): route replication read plan through boundary (#7448)

* test(scanner): cover flat-bucket quantum fairness (#7449)

Add a production-entry scanner cohort regression that combines a wide flat bucket with a small bucket under a fixed object budget. The test keeps partial budgeted rounds unpublished, verifies small buckets are only marked after real execution, and confirms a later unbudgeted round can publish the complete aggregate.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): reuse same admission request id (#7451)

Keep a retried heal start with the same request id from bypassing admission deduplication when the transport replay cache is unavailable but the manager still owns the task.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require perf summary evidence fields (#7453)

Fail closed when measured passing Scanner/Heal ABBA summaries omit W10/W11 foreground pressure, lock wait, or attempt-cost evidence.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): bind release evidence fields (#7452)

Require explicit scanner/heal release evidence field contracts for the scoped ACK and mixed-version rollback gates.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): tag dirty usage producer identities (#7454)

Record production-facing segment invalidation producer identities when existing object-level dirty usage hooks observe PUT, CopyObject, DeleteObject/DeleteMarker, and CompleteMultipartUpload mutations. Keep the data non-authoritative and process-local so segment reuse activation still requires durable generation-window proof.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): gate release evidence bundles (#7457)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require hard evidence ABBA manifest (#7459)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(heal): add MRF committed snapshot writer (#7458)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): cover EC8+4 restart shard rebuild (#7461)

Retry heal-control RPCs once after transport auth rejects a stale replay-scope epoch, and add a distributed EC8+4 restart heal evidence case that rebuilds a replaced drive with exact shard/body assertions.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): echo measured release evidence in ABBA fake adapter (#7460)

Keep the synthetic ABBA test adapter aligned with the measured release evidence contract so result validation covers release_evidence drift.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require complete ABBA summary matrix (#7462)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* heal: retain MRF replay journal after accepted/merged replays

Keep startup replay journal until a durable successor snapshot exists after Accepted/Merged admission.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* heal: reclaim superseded MRF snapshots after readback

Add a committed snapshot predecessor cleanup primitive that deletes only an older slot after the successor is read back as the current committed snapshot.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* heal: require exact owner for object receipts

Require object heal receipts to match the expected bucket incarnation before they can be recorded as positive repair evidence. This prevents stale or cross-incarnation receipts from clearing the wrong heal responsibility.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* heal: latch object receipt owner before repair

Capture the expected bucket incarnation before invoking object repair so a post-repair owner change cannot rewrite the responsibility that a storage receipt is allowed to prove.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* heal: replay committed MRF checkpoints durably (#7465)

Prefer committed MRF checkpoints during startup replay, retain accepted replay responsibilities until exact verified repair proofs arrive, and reclaim committed manifests only after discharge.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): bind hard evidence bundle provenance (#7467)

Require Scanner/Heal release bundle fields to carry source, run, window, timestamp, command, and artifact format provenance before a measured gate can pass.

Keep EC8+4 and performance gate fields tied to a single measurement window so unrelated artifacts cannot be stitched into a release approval.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* heal: reject dry-run object repair receipts

Do not record positive storage repair receipts for dry-run object heal tasks, even if a producer accidentally returns a matching receipt.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* heal: reject cancelled object repair receipts

Do not record positive storage repair receipts once an object heal task has been cancelled, even if the receipt still matches the requested owner and object identity.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): reject failed object repair receipts

Cover the receipt consumer path where a storage repair result carries both an error and a matching positive receipt. The failure may be recorded, but the receipt must not create repaired, healthy, or absent proof.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): cover bucket object repair receipts (#7468)

Cover bucket and root heal sweeps recording authoritative object outcomes only when storage receipts match the latched bucket incarnation.

Verify unavailable or stale receipt ownership keeps object repair execution intact while leaving canonical outcome proof as Unknown.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add EC8+4 heal restart evidence (#7469)

Register and wire Scanner/Heal background target restart and crash evidence cases for the 3x4 EC8+4 topology.

Validate the observed data/parity geometry in scanner-heal evidence receipts so multi-drive runs cannot satisfy the gate without proving EC8+4 metadata.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* heal: verify replacement pool metadata repair (#7471)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): cover quorum and mixed repair receipts

Add focused oracles for transient quorum results that carry a matching receipt and for mixed grace plus repaired receipt batches. Only the repaired object may produce positive proof.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* heal: verify admin recreate pool metadata (#7474)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* heal: preserve merged result for duplicate submits

Keep same-request-id replay receipts accepted for the receipt API, but preserve the legacy submit_heal_request duplicate admission result as Merged.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): preserve EC84 restart semantics (#7478)

Keep the EC8+4 background restart lane on the graceful-stop path and assert clean-restart marker absence only for restart scenarios. This prevents the hard evidence gate from silently exercising the crash path when it claims restart coverage.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): cover MRF snapshot torn successor recovery

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): write EC84 distributed restart oracle (#7481)

Bind the distributed EC8+4 restart evidence lane to a scanner/heal oracle artifact so release validation can consume the real nextest run instead of accepting only a passing test.

Require the registry to assert 8+4 erasure geometry for the three-node, four-drive case.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): harden MRF replay boundaries (#7483)

Reject journal records with unknown version-presence flags even when their CRC is valid, so rollback/future payloads cannot be accepted as known records.

Gate committed checkpoint cleanup by the writer owner captured from the replay source, preserving retained manifests from other owners inside the same sequence window.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): bind MRF manifest CAS dirsync recovery (#7482)

Cover the committed MRF manifest path through LocalDisk conditional CAS when the metadata directory fsync fails. The fixture proves the previous manifest anchor survives rollback, an unanchored first successor is removed, and the legacy MRF journal remains readable even while global durability is relaxed.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): derive evidence runner profile from registry (#7484)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require mixed-version evidence roles (#7485)

Tighten the Scanner/Heal release bundle checker so mixed-version, scoped-ACK, and rollback fields cannot reuse a generic versions list without proving the expected evidence role.

Require version lists to use source revision identities and include the tested source revision. Also require profile evidence fields to name the core profiling artifacts before release approval.

The release gate remains blocked until measured field evidence is present.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): publish committed MRF runtime checkpoints (#7490)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): bind profile artifacts in release evidence (#7487)

Require Scanner/Heal release bundles to attach every required profiling artifact to P1 profile evidence with relative paths, artifact formats, non-empty files, hashes, and optional per-artifact measurement-window checks.

Document the tightened release bundle profile contract and cover missing, tampered, and mismatched-window profile artifact regressions in the existing checker self-test.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): harden measured ABBA evidence claims (#7491)

Reject measured Scanner/Heal ABBA manifests whose mixed-version evidence uses the same baseline and candidate source revision or binary hash.

Require crash fault modes and profile artifact names to match the exact supported sets, rejecting missing, duplicate, and unknown values.

Update harness fixtures and regression coverage for same-build mixed-version claims and exact-set release evidence fields.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(e2e): group scanner heal evidence payload (#7494)

Group the EC8+4 Scanner/Heal evidence writer inputs into a typed payload so the distributed e2e crate stays within the clippy argument limit without weakening the lint.

The evidence writer still validates the same S3 bodies, physical shard census, process restart PIDs, and node listings.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require MRF replay bundle fields (#7486)

Bind Scanner/Heal release bundle evidence for MRF durable replay to replay counts, retained responsibility anchors, and successor snapshot publication evidence.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): profile EC84 evidence case runs (#7495)

Give Scanner/Heal evidence cases explicit runtime profiles so EC8+4 background restart and crash cases use their own object count, object size, and partial-progress timeout defaults instead of inheriting the legacy 4x1 case assumptions.

Expose the runtime profile in plan-only output and cover every registry case in the script self-test.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): reject empty release evidence artifacts (#7496)

Require Scanner/Heal release bundle artifact paths to resolve to non-empty files before hashing them.

Cover empty hard-gate artifacts in the existing release bundle checker self-test and keep profile artifact size checking on the shared artifact boundary.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require G14 same-window field coverage (#7489)

Tighten the Scanner/Heal release bundle checker so G14 same-window evidence must name the EC8+4, multi-set, and multi-pool fields covered in that measurement window.

Keep the release gate blocked when same-window evidence omits one of the required G14 fields, without changing production runtime behavior.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require two-hour measured ABBA windows (#7493)

Reject measured Scanner/Heal release ABBA manifests and summaries whose evidence window is shorter than the W21 two-hour release requirement.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): cover MRF idle checkpoint cleanup (#7497)

Add a runtime cleanup regression test that publishes both retained replay and runtime committed checkpoints, writes scoped and legacy journals, and verifies idle cleanup removes every recovery anchor from the registered local disks.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(e2e): ignore untracked files in build identity (#7498)

Align e2e_test build provenance with Scanner/Heal evidence receipts and server binary provenance by treating only tracked source changes as dirty.

This prevents unrelated untracked worktrees or evidence directories from causing compiled test identity mismatches before real evidence cases can run.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require MRF disk-full evidence fields (#7499)

Require the Scanner/Heal release registry and bundle checker to carry explicit G08 MRF capacity, disk-full, and replica-loss evidence fields before release approval can pass.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): bind release JSON artifact provenance (#7500)

Require scanner/heal release evidence JSON artifacts to repeat their measured source revision, run identity, measurement window, gate, and field identity inside the artifact payload. This keeps a refreshed outer bundle hash from accepting stale summary or profile JSON from another run.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): cover MRF rollback mirror filtering (#7501)

Add a regression oracle that keeps scoped-only MRF responsibilities in the authoritative runtime snapshot while omitting them from the v1 legacy rollback mirror.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require segment activation evidence (#7504)

* test(scanner): require segment activation evidence

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): cover activation proof inputs

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require MRF crash matrix cases (#7505)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require release bundle domain evidence (#7502)

Reject scanner/heal release bundles that omit field-level domain evidence for scoped ACK, durable intent, mixed-version, scheduler pressure, profile cost, and two-hour pressure lanes.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require MRF retention GC evidence (#7506)

Require P4 retained-responsibility release bundle evidence to list the retained replay anchor and idle cleanup cases, prove a two-hour retention window, and record both idle cleanup and verified-proof discharge observations.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require MRF cleanup GC soak evidence (#7507)

Require the P4 release bundle to carry measured cleanup/GC soak evidence for retained MRF replay responsibilities. The bundle now needs a two-hour cleanup window, exact cleanup case coverage, observed verified idle GC, and zero pending responsibilities or stale journals after GC.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): cover MRF disk-full commit anchors (#7509)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(e2e): report port allocator bind errors (#7510)

Surface the scanned port window, attempt count, and last bind error when the e2e port allocator cannot reserve a localhost port. This keeps Scanner/Heal evidence failures actionable when the environment blocks binds before business assertions run.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(sse): resolve 1.0.0 SSE/KMS blockers and P1 findings (#7511)

* fix(sse): resolve bucket default encryption per request

PUT and the POST-object/extract path resolved a bucket's default
encryption with a hard-coded "no explicit SSE-C" flag, so the default was
layered onto a request that already carried an SSE-C header triple and
then tripped that request's own mutual-exclusion check. Every bucket with
default encryption refused SSE-C single PUTs with 400 InvalidArgument,
while CreateMultipartUpload on the same bucket succeeded because it
resolves SSE elsewhere. Both call sites now derive the flag from the
request headers, as COPY already did.

The bucket default's KMS key id was also inherited independently of the
effective algorithm, so an explicit AES256 request against an aws:kms
default bucket produced a self-contradictory algorithm/key-id pair and
was rejected. The key id is now inherited only when the effective
algorithm is aws:kms, matching the storage-layer resolver.

Refs backlog#2368 B1, B2.

* fix(sse): refuse SSE-KMS without a running KMS service

A write requesting aws:kms on a node with no KMS service fell back to the
node-local SSE-S3 provider: the data key was wrapped with
RUSTFS_SSE_S3_MASTER_KEY while the object metadata still recorded
aws:kms and the requested KMS key id. The stored object claimed a KMS
protection it never had, under a key that was never consulted, and no
signal distinguished it from a genuine SSE-KMS object.

The managed-encryption path now asks the resolved DEK provider whether it
wraps with a node-local master key and refuses SSE-KMS in that case:
InvalidRequest when KMS was never configured, ServiceUnavailable when a
configured service is not running. The check sits after the per-key
authorization gate so an unauthorized caller still receives AccessDenied
whatever the KMS runtime state is, and asks the provider rather than a
parallel availability signal because the provider is what actually wraps
the key. A missing master key no longer answers an SSE-KMS request with
an SSE-S3-worded configuration error.

The SSE-S3 local fallback is unchanged.

Refs backlog#2368 B4.

* fix(ecstore): restore and archive tiers in stored coordinates

Multipart restore addressed the remote tier in plaintext coordinates
while the copy-back reads the stored representation. Each part received a
misaligned slice of the remote object whose length still satisfied the
range, the hash reader and the completion size check, so the restore
reported success and silently replaced the object's bytes. Encrypted and
compressed multipart objects were both affected. Restore now accumulates
stored part sizes, passes the stored length to the hash reader alongside
the plaintext length, and validates against the stored size.

The copy-back digests stored bytes, so its computed MD5 is not the
object's public ETag. Restore now preserves the object ETag on both the
single-part and multipart paths, and gives each restored part its own
recorded part ETag rather than the object-level value.

Transition also handed the tier the object's SSE headers and its
RustFS-wrapped data key as request headers. Any S3 target rejected an
SSE-C archive outright, an SSE-KMS archive asked the target to encrypt a
second time under a key id it does not own, and the wrapped DEK left the
cluster. The archive request now strips every SSE header and encryption
marker with the predicate the replication path already uses; the local
xl.meta keeps all of it, so read-through and restore are unaffected.

Objects restored by an affected release are not detected or repaired
retroactively and must be re-restored from the tier.

Refs backlog#2368 B3, B5; backlog#2369 P7.1.

* fix(rio): lock the v1 nonce layout within a segment

Decrypting a v1 segment tried three historical nonce layouts per frame,
independently for every frame. The last of them exists for streams
written before 1.0.0-alpha.91, which reused a segment's part nonce for
every block in it; because block zero's derived nonce equals that base
nonce, a frame encrypted at index zero authenticated at any position. An
attacker able to rewrite the underlying shards could replay it and have
the forged plaintext returned with 200 and an unchanged length. Shard
integrity uses a keyed-hash-free checksum, which such an attacker can
recompute, so it is not a barrier.

A segment now locks onto whichever layout decoded its first non-zero-index
frame and rejects any later frame needing a different one. That leaves one
residual shape: a stream built purely from repeats of frame zero has no
later frame to disagree. New RUSTFS_ENCRYPTION_LEGACY_NONCE_FALLBACK
(default true, so pre-alpha.91 objects keep decrypting) drops the third
layout entirely when set to false, which closes it. Turning it off refuses
pre-alpha.91 objects, so migrate them first by rewriting in place.

Refs backlog#2369 P2.

* fix(kms): reload a service that failed to start

POST /rustfs/admin/v3/kms/reload short-circuited whenever the persisted
configuration matched the in-memory one byte for byte. A node whose KMS
failed to start keeps that configuration and sits in Error, so the
documented recovery call returned "reloaded successfully" while leaving
the node down. Peers reached the same path through the reload broadcast,
so a cluster that lost Vault during a rolling restart had no working
recovery route other than the node-local start endpoint. Reload now
short-circuits only for a service that is actually running, and otherwise
reconfigures, which starts a service that is not running.

The AWS backend also advertised key-version enumeration through
kms/status, which its own documentation says it cannot do; the capability
and its golden snapshot now say false.

Refs backlog#2369 P1, P7.3.

* docs: record the SSE and KMS changes for 1.0.0

The Unreleased changelog section carried no entry for any encryption work
merged since 1.0.0-rc.5, including three items with operational impact:
the config-secret variable whose absence persists secrets in cleartext
with only a warning, the v2 frame write switch and its rolling-upgrade
constraint, and per-key authorization making a public bucket incompatible
with SSE-KMS objects. Adds those plus this batch, including the SSE-KMS
refusal as a breaking change with both routes out.

Also corrects four places where documentation contradicted the code: the
cleanup register still called encrypted range seek opt-in after its
default flipped, the Helm README claimed vault_mount_path only applies to
Transit while the template also feeds the KV2 mount, the disaster-recovery
drill listed bundle contents for backends whose export is refused with
501, and the Chinese README capability table predated most of the feature
set. Documents the SSE-S3 local master key as a first-class operational
mode with its rotation dead end, and what the v1 frame layout does and
does not authenticate.

Refs backlog#2369 P5.

* fix(kms): classify data-path KMS failures by what the caller can do

Only "key not found" and a backend outage were classified; every other
KMS failure that reached the S3 data path fell through to
500 InternalError with a generic message. A disabled or pending-deletion
key, a denied KMS grant, an encryption-context mismatch, an unsupported
algorithm, a credential or timeout failure, and a capability the
configured backend does not have all looked identical to a server fault.
SDKs therefore applied exponential backoff to configuration errors no
retry can fix, and monitoring counted every one of them against the
server's own error rate.

Unusable-key and request-side failures now answer 400, a denied grant
403, transient backend failures 503, and a missing backend capability
501. Damaged, unreadable, or unknown-format key material keeps its 500:
it is a server-side integrity fault, and existing tests pin it.

The classifier is deliberately separate from the admin lifecycle
mapping, which answers 404 for a missing key because there a key id is
the resource being addressed; on the data path it arrives inside a
request header or a bucket default. Messages either name what the caller
asked for or stay generic, with deployment-side detail left on the error
source the way the storage-IO mapping already does.

Refs backlog#2368 B6.

* fix(kms): track and renew static Vault tokens

Token authentication hard-coded "this token carries no lease", so the
renewal task never started, the remaining-TTL gauge was never published,
and nothing looked wrong. `vault token create` grants a 768-hour TTL by
default, so a cluster that had been healthy for a month turned every KMS
call into a 403 and could not recover without a restart or a
reconfigure. Production configuration validation only rejects the
literal dev-token, so an ordinary expiring token reaches a whole cluster.

The source now reads `auth/token/lookup-self` at login and adopts what
Vault reports. A token with no expiry behaves exactly as before. An
expiring renewable one is picked up by the existing renewal loop and
renewed at half TTL like every other auth method. An expiring
non-renewable one warns with its remaining lifetime and publishes the
gauge, so the fail-closed window is visible before it arrives.

The probe never fails the login: a policy that omits lookup-self, or a
Vault that is briefly unreachable, warns and falls back to exactly the
previous behaviour rather than taking down a deployment that works
today. The scripted Vault test double answers the lookup out of band so
existing scripts keep describing only the protocol under test.

Refs backlog#2369 P3.

* feat(sse): report SSE-C requests that arrive without TLS

An SSE-C request carries the customer's AES key in a request header, so
AWS S3 and MinIO both refuse one that did not arrive over TLS. RustFS
accepted them on any transport: a plaintext hop hands the key to anyone
on the path, and since the object cannot be read without that same key,
the exposure lasts as long as the object does.

Refusing outright is the correct end state but not a safe default to
adopt inside a release window, because the project's own s3-tests and
e2e lanes and most staging deployments speak plain HTTP. This release
reports instead: each such request increments
rustfs_ssec_plaintext_requests_total and logs one warning per process, so
an operator can confirm nothing would break before the default flips.
RUSTFS_SSE_C_REQUIRE_TLS=true opts into the AWS 400 now.

The verdict is per connection rather than per deployment: the layer is
built with whether this listener terminated TLS, and additionally accepts
an https protocol forwarded by a proxy the trusted-proxy configuration
already vetted. It sits beside the rate limiter, after the layer that
makes a forwarded protocol trustworthy and after the request context, so
a rejection can echo the request id.

Refs backlog#2369 P7.2.

* fix(kms): say what a node-local backend means for a cluster

The Local backend keeps key material on each node's own disk and
generates its Argon2id salt per node, so two nodes derive different keys
from the same master_key and an object encrypted on one node cannot be
decrypted on another. Behind a load balancer that surfaces as
intermittent 500s on reads that succeeded moments earlier, with nothing
tying the symptom to the cause: the only signal was a generic
"development, testing and demos only" positioning warning that says
nothing about what actually breaks.

Configuring or reconfiguring Local while the deployment is distributed
now logs a dedicated event and appends the consequence to the configure
response, so the operator who made the change sees it. The product
decision to warn rather than refuse is unchanged.

Refs backlog#2369 P7.4.

* docs: record the remaining SSE and KMS changes for 1.0.0

Adds changelog entries for the KMS data-path status classification, the
Vault static-token lease probe, the SSE-C plaintext-transport report and
its switch, and the node-local backend warning.

Documents two things the backend security guide never stated: that SSE-C
belongs on a secure transport, with the counter and switch to plan the
change around, and that the Local backend cannot be shared by a
multi-node deployment because each node derives different keys from the
same master key.

Refs backlog#2368 B6; backlog#2369 P3, P5, P7.2, P7.4.

* fix(kms): report an unreadable key store as an outage on the S3 path

A backend now distinguishes a key store it could not read from a key
that is genuinely absent, but the S3 boundary collapsed the first one
back onto 500 InternalError through the fallthrough for integrity
faults. The distinction was therefore invisible to the client: a
temporary key-directory outage looked exactly like a permanently damaged
key record, and neither the status nor the metric said the request was
worth retrying.

An unreadable key store joins the retryable class and answers 503, next
to a backend error and a credential failure. Damaged, unreadable or
unknown-format key material keeps its 500.

Refs backlog#2368 B6; builds on rustfs/rustfs#7470.

* test(storage): cover MRF cleanup delete ENOSPC anchor (#7513)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require heal retry stats evidence (#7514)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add release bundle dry-run fixture (#7515)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): confirm recovery intent accept readback (#7516)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require MRF cleanup delete ENOSPC evidence (#7518)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): require bounded retry window evidence (#7517)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(e2e): emit scanner heal G09 upgrade evidence (#7519)

Record measured Scanner/Heal G09 evidence artifacts from the upgrade compatibility lanes when a fresh evidence directory is provided.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(heal): persist MRF rollback mirror boundary (#7520)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add G09 upgrade evidence runner (#7522)

* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* test(scanner): add G09 upgrade evidence runner

Add a reusable Linux x86_64 runner for the Scanner/Heal G09 mixed-version and rollback upgrade evidence lanes.

The helper reads the pinned previous-release asset metadata from the upgrade workflow, verifies the downloaded binary, builds the current head, runs both ignored E2E tests, and fails unless the expected G09 JSON artifacts exist.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): implement V2 evidence preflights (#7523)

* feat(scanner): wire dirty usage producer identities

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add segment activation preflight proof

Keep scanner segment reuse behind a structured activation preflight so release evidence can prove the production gate remains disabled until every producer, generation, overflow, cold-oracle, and distributed invalidation check is satisfied.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): expose distributed invalidation evidence

Record an explicit distributed segment invalidation evidence summary when remote dirty usage snapshots are bound to the current activity window and the authenticated scoped ACK capability probe succeeds.

Reject peer dirty usage snapshots that contradict the peer activity pending bit so scoped ACKs fail closed instead of clearing an unadvertised remote mutation.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): assemble release evidence bundles (#7526)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): add G09 upgrade evidence runner (#7524)

* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics and therefore proved nothing about the filter this fix
changed. It now also carries a two-tag `And` rule - the shape
`mc replicate add --tags "k1=v1&k2=v2"` writes - and asserts that an object
with one of the two tags is not admitted while an object with both is.

No new test function, so the nightly selection digest is unchanged.

* refactor(site-replication): fold the refresh state-change error into one constructor

The endpoint-refresh work added three `s3_error!` invocation lines, which the
s3s footprint ratchet is meant to prevent. Five copies of the same
concurrent-change error now share one constructor, so the surface nets one
line smaller than main; the baseline is retightened to match.

* fix(site-replication): report a peer whose IAM snapshot waits for a repair

An escalated snapshot entry records a deletion a snapshot cannot replay, so
only a repair settles it and the marker must survive. Scheduling an import
snapshot therefore leaves that peer's entry alone - and now says so, instead
of returning success while nothing was scheduled for it.

* docs(operations): state the group-status and escalation convergence limits

Two boundaries the fixes in this branch make load-bearing: a membership change
never carries an enable, so a group disabled on one site only has to be
re-enabled there explicitly; and a peer holding an escalated IAM entry does
not receive a scheduled snapshot, including the one a bulk import schedules,
until a repair settles it.

* fix(ci): bind performance runs to selected inputs (#7512)

* test(scanner): add G09 upgrade evidence runner

Add a reusable Scanner/Heal G09 runner for Linux PR-head validation. The script downloads the pinned previous release, builds the current checkout, runs the mixed-version and rollback upgrade E2E lanes, and validates the measured raw artifacts before they can be consumed by the release bundle gate.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): replay recovery intents while disabled (#7521)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): preflight G09 evidence disk space (#7527)

* fix(replication): close the GA blocker set from backlog#2366 (#7503)

* fix(replication): close GA blockers from backlog#2366

Implements the P1 set from the pre-GA replication audit:

- Replication rule tag filters now require every And.Tag to match, replacing
  the s3s OR semantics with a local AND matcher that fails closed on a
  malformed tag.
- A replicated group membership change no longer writes the group status, so
  a membership update carrying the default Enabled status cannot silently
  re-enable a disabled group on the peer.
- A successful IAM import schedules one collapsed full-IAM snapshot per remote
  peer instead of leaving the imported entities local-only.
- A pending endpoint refresh is redriven by the heavyweight reconcile tick,
  carries its own ilm-expiry override, and no longer blocks a remove that
  drops every unacknowledged peer.
- Site metrics expose local replication failure totals and rolling windows;
  node-level counters no longer report a constructed zero.
- set/remove-remote-target notify peer metadata caches before returning, so a
  follow-up put-bucket-replication on another node sees the target.
- Adds the site-replication operations runbook, a docs index, a replication
  support boundary section, and the Replication changelog section.

* fix(site-replication): resume only a locally driven endpoint refresh

The peer-side edit handler journals a pending endpoint refresh with an empty
`remote_peers` map and commits it inside the same request through
`apply_internal_peer_edit`. The reconcile tick could not tell that journal
from the coordinator's own: with no required peers it reads as complete on
sight, so the tick committed it with `edit_state` - losing the local-name
sync - and cleared it under the request that owned it, whose commit then
reported the refresh as changed and denied the coordinator the peer
acknowledgement it was waiting for.

Resume now runs only for a journal that carries the fan-out topology. A
receiver's journal stays for the coordinator to redrive with the same
refresh id, which is the path that already recovers it.

* fix(site-replication): keep an explicit disabled group status on a snapshot

Skipping the group-status write whenever an item carries members stopped a
membership change from re-enabling a disabled group, but it also silenced the
full-IAM snapshot, which always sends members together with the sender's real
status. A peer that did not have the group yet created it through
`GroupInfo::new` - enabled - so a bootstrap, a repair, or the snapshot an IAM
import now schedules handed every member of a frozen group live access there.

The madmin wire maps an unset `groupStatus` to Enabled, so only Enabled can be
a default. Disabled is always explicit and is applied again.

* fix(site-replication): schedule the import snapshot without recording a failure

`import-iam` reused the failure-recording path to queue its full-IAM
snapshot. That raises `retry_count` on every call, so three imports - the
normal shape of a bulk migration done one archive at a time - escalated a
healthy peer to `retryStats.failed` with the scheduling note shown as
`lastError`, which is exactly the signal the runbook tells operators to
repair. A full retry queue also turned a completed import into a 503.

Scheduling now only ensures the collapsed entry exists, and a failure to
schedule is logged instead of failing the request: the entities are already
imported and the reconcile pass still closes the gap.

* fix(admin): stop reporting replication failures as retries

`retries` is the minio-go counter for redeliveries, and mc prints it as such.
Filling it with the failure count claimed a redelivery that never happens: a
failed object is not retried by an event today, it waits for the scanner heal
pass. `errors` keeps the failure counters; `retries` stays zero until there is
a real redelivery to count, and the runbook now says so.

* perf(site-replication): aggregate failure windows without cloning bucket stats

`site_metrics_snapshot` went through `get_all`, which clones every bucket's
stats, and then scanned each target's sample deque twice. That deque is
bounded only by the one-hour window, so an unreachable target under load -
the case an operator polls this endpoint for - made every
`mc admin replicate status` copy the whole backlog and hold the read lock
against the failure path while doing it.

It now folds under the read lock and takes both windows in one walk. The
`max` against the serialized `last_minute` / `last_hour` snapshots is dropped:
those are stamped onto per-bucket clones elsewhere and are always zero in this
node-local cache.

* fix(site-replication): reject a conflicting ilm-expiry override on a re-run

The commit now reads the ilm-expiry override back out of the pending refresh
journal, so a second edit that asks for a different value had it dropped while
the request still reported success. Re-running without the flag keeps pinning
the recorded value - that is the documented way to redrive a stuck refresh -
but an explicit different value is now rejected instead of ignored.

* fix(admin): do not fail a remote-target write on a peer reload error

set/remove-remote-target propagated the peer metadata reload error, so a
target that was already persisted and live on this node reported a 5xx to the
client whenever one peer could not be reached. Every S3 bucket-config write
path treats that reload as best effort and only warns; these two admin
handlers now do the same, and the reason is logged with the bucket and action.

* fix(site-replication): undo every bucket a cut-short refresh rewrote

When a remove accepted on another node clears the refresh journal mid-pass,
only the bucket holding the lock at that moment had its restored target
undone. The buckets rewritten earlier in the same pass kept a target pointing
at the removed peer whenever the remove's own cleanup had already walked past
them. The undo now covers every bucket this pass rewrote, attempting all of
them so one failure does not strand the rest.

* fix(site-replication): keep replay running while an endpoint refresh is pending

A pending endpoint refresh took the whole heavyweight pass with it, so a peer
that never came back froze IAM and bucket replay to every healthy peer too -
the stall this journal's resume path was meant to end. The refresh arm now
drains the retry queue before returning; it replays per-peer deliveries
against the endpoints currently committed in state, so it is unaffected by the
edit in flight. Bucket wiring reconciliation still waits, because it rewrites
the very targets the refresh is changing, and the runbook now says so.

* test(e2e): cover the AND semantics of a two-tag replication filter

The acceptance matrix only had a single-tag rule, which matches under both AND
and OR semantics…

* fix(list): reconcile split overwrite metadata (#7910)

fix(list): fail closed on split overwrite metadata

* fix(heal): persist explicit delete-marker purges (#7908)

* fix(heal): persist explicit delete-marker purges

* fix(ecstore): remove unused object info wrapper

* test(rpc): use valid incarnation in digest gate

---------

Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>

* ci(table): pass --outcome to the backlog issue manager (#7914)

The manager step failed with 'the following arguments are required:
--outcome' in the first table run (hidden by continue-on-error): every
other suite workflow passes the test step outcome through.

* fix(list): reconcile encoded directory objects (#7915)

* fix(ci): restore release checks after marker purge changes (#7918)

* fix(rpc): recover tier reloads after peer capability revocation (#7920)

* test(ecstore): drain expiry workers before restart (#7923)

* test: report tier cleanup recovery progress on timeout (#7922)

* fix(heal): resume replacement generation after target restart (#7926)

* ci(release): use sm-standard-4 for Linux builds (#7925)

* feat(release): add workspace cargo publish tooling

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(release): prepare 1.0.1 metadata

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix

* revert

---------

Signed-off-by: houseme <housemecn@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: 唐小鸭 <tangtang1251@qq.com>
Co-authored-by: cui fliter <imcusg@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: xiaomage <devops008@sina.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: hehutu <hehutu@gmail.com>
Co-authored-by: Henry Guo <marshawcoco@gmail.com>
Co-authored-by: RJ Regenold <rregenold@teamraft.com>
Co-authored-by: RJ Regenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: Jason Kossis <jkossis@gmail.com>
Co-authored-by: Hiroaki KAWAI <1468181+hkwi@users.noreply.github.com>
Co-authored-by: Hiroaki KAWAI <hiroaki.kawai@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-17 18:15:35 +08:00
cxymds a58ac62618 fix(rebalance): defer retryable source cleanup failures (#7977)
Source cleanup treated a retryable conflict such as a lock acquisition
timeout as a permanent cleanup warning. The warning was persisted, mapped
the pool to failed_ignored, and blocked pool completion for the rest of
the run while the source replica stayed on disk. Retryable storage
failures now defer the bucket instead, and only non-retryable failures
keep a permanent warning, so completion still requires a resolved delete
or a successful not-found check.

Recording a bucket deferral no longer overwrites the pool's pending
migration-deferral marker, because that marker is the only signal keeping
a pool from completing at the free-space goal while an entry is retried.

Teach the transient matchers the rendered lock timeout text, unwrap
data-movement stage context before classifying not-found and transient
cleanup failures, and raise the bounded source cleanup deferral budget.
2026-09-17 10:14:54 +00:00
Hauser 36c6c2e34b fix(lock): contain slow peer RPC admission storms (#7970)
Add per-peer acquisition backoff, degraded in-flight admission, deadline propagation, bounded late-release cleanup, and rate-limited diagnostics for slow lock endpoints. Preserve quorum and fail-closed held-lock refresh semantics.

Refs: rustfs/backlog#2579

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-17 10:01:16 +00:00
Hauser d6729a1138 fix(heal): prevent MRF starvation of integrity repairs (#7963)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-17 08:18:26 +00:00
Hauser 07a9d15ab2 fix(readiness): smooth pool metadata timeout flaps (#7956)
Keep recent writable observations during transient pool metadata inspection timeouts while preserving confirmed write blocks and fail-closed behavior. Consolidate the runtime observation cache into one ordered state machine and expose timeout metrics.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-17 06:53:55 +00:00
Chris d9c3eeea0c fix(ecstore): hide and reclaim ancestors of delete residue in prefix listings (#7955)
* fix(ecstore): hide ancestors of delete residue in prefix listings

Since #7342 the never-versioned fast path hides a directory whose children are all deleted-version data dirs, but only that one level: the date-style ancestors above a deleted key (`metrics/<stream>/2026/08/28/23/`) still surfaced as empty folders that nothing could remove, which is what #6898 keeps reporting on rc.6 and 1.0.0. The residue probe now walks down through metadata-less ancestors depth-first, ending at the first file met outside a UUID data dir, with a total read budget so a large residue tree surfaces and is hidden one level down instead of costing an unbounded walk. An empty delimiter listing of the ancestor then reclaims the whole committed residue tree in one pass.

Refs #6898

* fix(ecstore): sample before full reads in the residue probe and purge orphan subtrees independently

Adversarial pass 1 findings: the probe read a wide directory in full at every level of a genuine prefix once its 8-entry batch was full, so a leaf holding thousands of objects was materialised per listed prefix; the probe now descends into the sampled children first and only reads the remainder once every sample proved unlistable, and a test pins one bounded read per level and zero complete reads on a wide genuine prefix. Hiding ancestors also made purge_orphan_dir_object fail closed on a whole ancestor when committed residue shared it with residue an older build left without markers, which 1.0.0 users could still reclaim by browsing one level lower; the purge now blocks only the ancestor chain of unpurgeable data and reclaims purgeable sibling subtrees. A UUID directory holding a subdirectory surfaces again as before.

Refs #6898

* fix(ecstore): back off repeated orphan purge scans of unpurgeable trees

Adversarial pass 2: with ancestors hidden, every empty listing of a phantom prefix scanned the whole residue subtree on every disk even when nothing under it can be purged (no committed marker), and the scan no longer stops at the first blocked directory. Remember such prefixes per set for 60 s and skip the rescan. Build committed residue file paths like directory paths so the blocked filter matches under keys containing repeated slashes.

Refs #6898
2026-09-17 10:11:45 +08:00
Aria Stewart 892d3ba3f7 Allow 127.0.0.0/8 as loopback addresses for binding (#7943)
This is particularly useful for integration testing of services that use
an S3 API, as it lets many different instances bind a port on different
IPs.

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-17 10:11:36 +08:00
464 changed files with 50264 additions and 5785 deletions
+39 -25
View File
@@ -1,6 +1,6 @@
---
name: rustfs-release-publish
description: "Run the end-to-end RustFS console gate, version bump, preview validation, human confirmation, and final-tag publication pipeline. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
description: "Run the RustFS console gate, source version bump, release-branch CI, preview validation, human confirmation, final-tag publication, and post-release installation and website updates. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
---
# RustFS Release Publish (preview-validated pipeline)
@@ -18,16 +18,20 @@ Pipeline shape:
```
check console main against its latest Release
-> if ahead: publish console -> wait for Release asset + latest API
-> bump RustFS version files to <target> (final version, ONE commit) -> merge
-> tag <preview-tag> at that commit -> CI green
-> bump Cargo.toml and Cargo.lock to <target> -> merge
-> cut release/<target> from a selected main commit containing the bump
-> CI green for that exact release-branch commit
-> tag <preview-tag> at that commit
-> verify preview Release assets -> run binary locally + console checks
-> validate with latest rc client
-> report preview acceptance results -> STOP for explicit human confirmation
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
-> CI deletes the <target>-preview.N Releases (tags kept)
-> verify published images -> publish Helm chart from the validated source
-> update installation references on main -> update rustfs.com announcement
```
On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
On candidate or preview validation failure: fix lands on main via normal PR, then backport only the needed fix to `release/<target>` through a reviewed PR. Re-run release-branch CI at the new commit and, if a preview was already tagged, tag `<preview-tag N+1>` there and restart acceptance. Installation references remain on the previous published deliverable throughout preview validation.
## Required inputs
@@ -56,22 +60,24 @@ Rules:
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
## Hard rules
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
- Before preview, bump only Cargo.toml (workspace package and internal dependency versions) and the corresponding workspace members in Cargo.lock, directly to `<target>`. Keep README installation examples, flake.nix, Chart.yaml, and rustfs.spec on the previous published deliverable until Phase 7. Never write a `-preview.N` suffix into these files; preview identifiers belong to validation tags and artifacts.
- Publishing a tag alone does not make an installation target available. Verify the exact image manifest, Release assets, and source archive before advancing references that consume them. Helm packaging derives its versions from the final tag and waits for that image; it does not require an early Chart.yaml bump on main.
- Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
- Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
- The release branch is the candidate source during multi-day acceptance. Do not merge main wholesale into it, force-push it, or advance it with unrelated changes. The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD or an advanced release-branch HEAD, and never create an extra version-bump commit between preview and final. Phase 7 updates installation references on main after publication; neither tag moves to that follow-up commit.
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
- Phases run in order; a failure blocks dependent steps. For a candidate or preview/source acceptance failure before the final tag exists, land the fix on main, backport it to the release branch, verify Cargo there still matches the confirmed target, and restart from release-branch CI in Phase 2. Use the next preview iteration if any preview was already tagged. Main may advance to another target without changing this release candidate.
- Once the final tag exists, preserve its commit. Retry failed publication jobs for that exact tag; do not recreate the tag or restart preview on newer main. If a source change is required after final-tag publication, report the failure and obtain a new target version. Phase 7 failures resume only the failed follow-up after rechecking artifact availability and whether a newer release has superseded it.
- Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
- Confirmation is scoped to the reported `<target>`, `<preview-tag>`, and `PREVIEW_HASH`. A failed or repeated acceptance cycle, including any new preview iteration, invalidates prior confirmation and requires a new one.
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision. Do not delete the release branch or preview tags as part of abandonment without an explicit decision.
- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
## Phase 0 — Preflight
@@ -79,25 +85,27 @@ Rules:
- `git status --short` clean; `git fetch origin main --tags`.
- `gh auth status` works; confirm you can view `gh release list -L 3`.
- Confirm the exact final target version with the user if not explicit.
- Check whether `release/<target>` already exists before creating it; a resumed release reuses the existing branch and its history. For an in-flight release that already has a preview tag but no branch, anchor the new branch to that tag's commit only after verifying its target versions and prior acceptance evidence.
### Console release gate
Read and complete [the Console gate](references/console-gate.md) before Phase 1. Verify the latest published Console asset and exact commit; if Console main is ahead, complete its release and asset verification first. A successful build alone does not satisfy this gate.
## Phase 1 — Version bump to the final target (once)
## Phase 1 — Source version bump to the final target (once)
- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
- Get the PR merged into main. Record the resulting main commit:
- If `release/<target>` exists, verify its Cargo.toml and workspace members in Cargo.lock read `<target>` and skip the source bump, even if main has advanced. If the branch versions differ, stop and resolve the mismatch without downgrading main.
- If no branch exists but an unfinished preview tag does, verify its commit has `<target>` in both version files and skip the bump; Phase 2 anchors the branch to that commit. Stop if those files disagree. Otherwise, if main already has `<target>` in both files, skip the bump. Installation references intentionally retain the previous published deliverable during preview; if they were advanced to an unavailable target, restore that baseline before choosing the next candidate.
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
```bash
git fetch origin main
PREVIEW_HASH=$(git rev-parse origin/main) # must contain the bump PR
```
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
## Phase 2 — Publish the preview tag
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
@@ -106,17 +114,18 @@ git push origin "<preview-tag>"
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
## Phase 3 — CI and preview Release verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
## Phases 4–5 — Local artifact, Console, and rc acceptance
@@ -124,13 +133,13 @@ Read and complete [preview acceptance](references/preview-acceptance.md): verify
### Manual confirmation gate
After every Phase 3–5 check passes, report the target, preview tag, `PREVIEW_HASH`, preview Release URL, console result, and rc matrix, then explicitly ask the user whether to publish the final tag. End the turn without creating or pushing `<target>`.
After every Phase 3–5 check passes, verify the release branch still points to `PREVIEW_HASH`. Report the target, release branch, preview tag, `PREVIEW_HASH`, release-branch CI run, preview Release URL, console result, and rc matrix, then explicitly ask the user whether to publish the final tag. End the turn without creating or pushing `<target>`.
Continue to Phase 6 only after a new user reply explicitly confirms the reported target, preview tag, and commit. A clear affirmative reply to that exact report, such as `确认继续`, is sufficient; if the reply is ambiguous or any reported value changed, ask again.
## Phase 6 — Publish the final tag on the validated commit
No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH`. If it moved after confirmation, the approval no longer applies; validate and obtain confirmation for a new preview. The final tag goes on the exact commit the preview validated:
```bash
git fetch origin --tags
@@ -145,12 +154,17 @@ git push origin "<target>"
- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
## Phase 7 — Installation references and website announcement
Only after Phase 6 succeeds, complete [post-release updates](references/post-release-updates.md): align installation references on main, then update the existing top banner in `rustfs/rustfs.com`. A failed or incomplete publication leaves both on the previous available version. These follow-up commits never change `PREVIEW_HASH` or either release tag. Track their PR, merge, and deployment states separately from artifact publication; an open PR is not a live website update.
## Output contract
Always report:
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
- Target version, release branch, release-branch CI run and validated SHA, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
- Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
- Post-release installation PR and verification results; website banner target, text, link, PR, and observed deployment state. Report any remaining merge authorization or failed deployment explicitly rather than claiming the banner is live.
- Any deviation from this pipeline and why the user approved it.
@@ -0,0 +1,32 @@
# Post-release Installation and Website Updates
Read only after the final non-preview tag has passed Phase 6. Tag creation or a green binary build alone is insufficient: the release assets, source archive, container images, Helm package, and applicable latest channels must be available. Preview tags never enter this phase.
## Installation references
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
## rustfs.com top banner
Use the existing announcement configuration from [website PR #112](https://github.com/rustfs/rustfs.com/pull/112). Do not add a second banner or redesign the header.
1. In an isolated checkout of `rustfs/rustfs.com`, read its current `AGENTS.md`, PR template if present, `data/announcement.ts`, and the consuming header component. Search for an existing announcement PR for this target before creating another.
2. Update `homeAnnouncement` in `data/announcement.ts` after the release and installation availability checks pass. Keep `enabled: true`, `badge: 'New'`, and the existing component. For a stable release, use:
```ts
export const homeAnnouncement = {
enabled: true,
badge: 'New',
message: 'RustFS <target> is now available! Please',
linkText: 'upgrade and try',
href: '/download',
} as const
```
Replace `<target>` with the exact published version. Reserve GA wording for the actual first stable release. For a final alpha/beta/rc tag, explicitly call it a prerelease; use its GitHub Release URL if `/download` does not expose that target. Never announce a `-preview.N` build here.
3. Use the site's current package manager and required checks. For this config-only change, type-check and lint the changed file, build the site, and inspect the generated homepage for the version, visible link text, and destination. Confirm `/download` actually offers the announced version before using that link.
4. Commit, push, and create or update the website PR when included in the authorized release scope. Include the published RustFS Release URL and checks run. Keep merge and production-deployment authority separate; reuse permission already given, but this reference grants none by itself.
5. After an authorized merge/deployment, verify the live `https://rustfs.com/` banner and its destination. If only the PR is ready, report that state and URL; do not report the banner as deployed. A website failure does not invalidate or move the published release tag: retry this phase after resolving the failure.
@@ -1,16 +1,15 @@
---
name: rustfs-release-version-bump
description: "Prepare the version-file and release-asset bump for an exact RustFS alpha/beta/stable target, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
description: "Prepare Cargo versions for an exact RustFS target, or align installation references after its artifacts are published, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
---
# RustFS Release Version Bump
Use this skill to prepare and verify release version files. Commit, push, and PR steps apply only when included in the user's delivery scope; publishing release tags belongs to `rustfs-release-publish`.
Validated baseline: release pattern used in PR `#2957`.
## Required inputs
- Exact target version, for example `1.0.0-beta.4`.
- Stage: `prepare` (default) or `post-release`. Infer `post-release` only when the user or parent release workflow explicitly requests installation updates after publication; a normal version bump means `prepare`.
- Delivery scope: local (`edit/verify`), git (`commit/push`), or GitHub
(`commit/push/PR`). Derive it from the conversation; when unspecified, prepare
and verify locally without blocking on a delivery question.
@@ -25,25 +24,32 @@ Reject any target version containing `-preview`: preview identifiers are tag-onl
- `.github/pull_request_template.md` only when preparing a PR.
- Current branch status and diff against `origin/main`.
## Default release file scope
## Stage boundaries
Treat the following file list as the default checklist for each release bump:
`prepare` updates only:
- `Cargo.toml`
- `Cargo.lock`
`post-release` updates installation and packaging references only:
- `README.md`
- `README_ZH.md`
- `flake.nix`
- `helm/rustfs/Chart.yaml`
- `rustfs.spec`
Only drop a file when the current repository release process clearly no longer requires it.
During preview, installation references intentionally retain the previous published deliverable. Do not sweep them into a source bump to make every version string match. `flake.nix` builds local source, but its package label is aligned with the other packaging references after publication.
Before `post-release` edits, verify that the exact non-preview target has a published GitHub Release, downloadable assets and source archive, and a pullable `rustfs/rustfs:<target>` image manifest. If any prerequisite is missing, report `BLOCKED` and leave installation references unchanged. If a newer deliverable has already superseded this target, do not downgrade installation defaults during a retry.
Helm CI derives chart versions from the triggering tag, so the final tag can retain the previous Chart.yaml values. Update the repository copy only after the target image and published chart are available. Neither post-release changes nor their merge commit may replace the preview-validated final tag.
## Hard release policy
- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
- Helm chart version mapping follows `beta.N -> 0.N.0`.
- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
- Derive Helm chart and app versions with `scripts/helm_chart_version.sh <target>`; the existing mapping is `beta.N -> 0.N.0`, with other target versions retained.
- `rustfs.spec` `Release` uses the prerelease suffix (for example `beta.4`), or `1` for a stable release.
- Do not change these rules without explicit confirmation.
## Step-by-step workflow
@@ -52,22 +58,18 @@ Only drop a file when the current repository release process clearly no longer r
- Use the exact target and delivery scope already supplied; ask only for a missing or ambiguous target or a material release-policy choice.
- Inspect current branch and ensure only release-related files are touched for this task.
2. Update workspace versions
2. Update workspace versions (`prepare` only)
- Bump `[workspace.package].version` in `Cargo.toml`.
- Bump internal workspace crate dependency versions in `Cargo.toml`.
- Update `Cargo.lock` so workspace package versions match target version.
- Re-scan for partial leftovers.
- Re-scan Cargo.toml and workspace members in Cargo.lock for partial leftovers; leave external dependency versions unchanged.
3. Update release assets
3. Update installation references (`post-release` only)
- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
- `flake.nix`: update package version to target version.
- `helm/rustfs/Chart.yaml`:
- `appVersion` = target version.
- `version` follows chart mapping rule, for example:
- `1.0.0-beta.3` -> `0.3.0`
- `1.0.0-beta.4` -> `0.4.0`
- `helm/rustfs/Chart.yaml`: use the app and chart versions returned by `scripts/helm_chart_version.sh <target>`.
- `rustfs.spec`:
- Set `Release` to prerelease suffix (example `beta.4`).
- Set `Version` to the numeric version and `Release` to the prerelease suffix (example `beta.4`), or `1` for a stable release. Verify that `Source0` and the unpacked source directory resolve to the exact published target, including its prerelease suffix when present.
- Add/update top changelog entry with exact format:
- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
- `- Update RPM package to RustFS 1.0.0-beta.4`
@@ -78,15 +80,14 @@ Only drop a file when the current repository release process clearly no longer r
- Changelog version text must match target release version exactly.
4. Verify before shipping
- Run:
- `make pre-commit`
- If `make pre-commit` fails, fix task-attributable failures and rerun affected checks. Report unresolved required checks as `BLOCKED`; do not silently widen scope to fix unrelated issues.
- Follow the root verification tiers for the final diff instead of running a full-workspace gate for version strings.
- For `prepare`, validate Cargo metadata with the updated lockfile and confirm workspace package/internal dependency versions agree. Do not accept a lockfile containing unrelated dependency updates.
- For `post-release`, render the Helm chart and confirm its default image is the verified target; run `scripts/test_helm_chart_version.sh` when chart versions change. Check the README image tags, Nix package version, and expanded RPM source URL against that same target.
- Run `git diff --check` for either stage. Report unresolved required checks as `BLOCKED`; do not silently widen scope to fix unrelated issues.
5. Commit strategy (only when committing is authorized)
- Preferred split when both parts changed:
- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
- `chore(release): align release assets for <version>` for docs and packaging files.
- If user asks for one commit, use one commit.
- Use `chore(release): prepare <version>` for `prepare` and `chore(release): align installation references for <version>` for `post-release`.
- These stages happen on opposite sides of publication; do not combine them in a preview-preparation commit or PR.
- Stage only intended release files; do not include unrelated working tree changes.
6. Push and PR (only for the authorized delivery scope)
@@ -104,13 +105,12 @@ Only drop a file when the current repository release process clearly no longer r
- `git diff --name-only origin/main...HEAD`
- `git diff --stat origin/main...HEAD`
- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
- `make pre-commit`
## Output contract
When using this skill, always report:
- Target version.
- Target version and stage.
- Files changed.
- Any assumptions or uncertainties requiring confirmation.
- Verification result (`PASSED` or `BLOCKED`) with key evidence.
@@ -101,6 +101,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
### SSE and on-disk storage invariants
- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
- `GHSA-wqmc-vjgv-jrpw`: SSE-C persisted and listed the plaintext MD5 as the ETag, allowing list-only users to confirm or brute-force low-entropy content and correlate equal plaintext across keys. Lesson: encrypted-object ETags must not expose deterministic plaintext fingerprints; protect them with per-object key material and keep listing, conditional, copy, and multipart semantics consistent.
### Object Lock and retention invariants
@@ -126,7 +127,7 @@ rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonical
rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
rg -n "HashReader|EncryptReader|try_resolve_etag|ETag|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
```
@@ -145,5 +146,5 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-linux=21ab66fd172018682ce825a53f26aa8f286d57f3f687d8616d1850e56474dcd8
sha256-darwin=21ab66fd172018682ce825a53f26aa8f286d57f3f687d8616d1850e56474dcd8
sha256-linux=6fc377fa1f9f06e065f077c7bd53d04efbccf4453f95dde67afaa47778185377
sha256-darwin=6fc377fa1f9f06e065f077c7bd53d04efbccf4453f95dde67afaa47778185377
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=9e9687ded961aa4e619a15def6e1ee0a5153be05423d009ee7a58126624c83e7
sha256-linux=364666693c2bf258997643c9d87e07e7b7501e58175f190d201d53c81946eb20
sha256-darwin=6b8e35ef69456bd244d6a6b19179e408763763e1e163aaaf90055490ae12da4b
sha256-linux=ff4c40f288ab5e91fd28d533c277538c0e3281fa353328ab52f5a8f88e0dc931
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=83a7dcaffd5a789517ae9f02a224f66a9713937885cff96fca2ad7e216f197ae
sha256-linux=626c10f8c964507ff987b6c86069e9019dc6d2ae7fb02db9be5df5aa8cc5145b
sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1
+1 -1
View File
@@ -1 +1 @@
sha256=548a58d74f3cb3b5a1ab9dcd4d3f8625b9ddebe847a2e14de1d08c3b2e6292eb
sha256=7e3989625e2e6087c2e7db7ee02d30624b566d5a99832122032fb413ff456662
+4
View File
@@ -24,11 +24,13 @@ TEST_THREADS ?= 1
.PHONY: script-tests
script-tests: ## Run shell script tests
@echo "Running script tests..."
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
./scripts/test_build_rustfs_options.sh
./scripts/test_docker_runtime_timezone.sh
./scripts/test_entrypoint_credentials.sh
./scripts/test_internode_grpc_ab_bench.sh
./scripts/test_object_batch_bench_enhanced.sh
./scripts/test_package_service_scripts.sh
./scripts/test_hotpath_warp_ab_gate.sh
./scripts/test_hotpath_warp_abba.sh
./scripts/test_scanner_validation_harness.sh
@@ -50,6 +52,7 @@ script-tests: ## Run shell script tests
./scripts/check_embedded_secrets.sh --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_scheduled_validation_freshness.py --self-test
@@ -59,6 +62,7 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
$(RUSTFS_PYTHON_BIN) ./scripts/check_object_data_cache_follower_samples.py --self-test
./scripts/validate_object_data_cache_cold_stampede.sh --self-test
+105 -15
View File
@@ -82,6 +82,13 @@ setup = 'lifecycle-large-stack'
filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
setup = 'ecstore-large-stack'
# These real-disk scanner fixtures assert quiet-workload publication and
# durable MRF admission. Reserve the pool while their bounded CAS/fsync
# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
[[profile.default.overrides]]
filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
threads-required = "num-test-threads"
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
test-group = 'ecstore-serial-flaky'
@@ -107,6 +114,16 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
test-group = 'ecstore-serial-flaky'
# Serialize the replacement bucket metadata heal test. It builds a real-disk
# hermetic erasure set, writes bucket metadata, then asserts byte-identical
# xl.meta readback after a rejected delayed repair. Under parallel load,
# concurrent ecstore tests mutate the process-wide disk registry and alter
# the serialized timestamp/disk-identity fields, flipping the assertion.
# Preventive serialization only, no retries.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
test-group = 'ecstore-serial-flaky'
# Serialize the metadata-cache generation-retirement pair. Both carry
# #[serial(metadata_cache_invalidation_probe)] — a no-op across nextest's
# process boundary — and assert get_object_metadata_cache generation
@@ -132,10 +149,37 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
test-group = 'embedded-test-ports'
# The real object probe includes bucket creation and cleanup in its short
# measurement budget. Keep competing storage fixtures outside that budget.
# Registration, inventory delivery, and real drive/object probes include
# durable filesystem IO in short deadlines. Reserve capacity so sibling test
# processes cannot exhaust those budgets. Keep deadlines and assertions intact.
[[profile.default.overrides]]
filter = 'package(rustfs) & binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
threads-required = "num-test-threads"
# This cancellation probe hashes the serving executable before acquiring its
# collector lease. Keep competing test processes out of its 30s startup budget.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(=connect::diagnostics::trace_runtime::tests::local_runtime_profile_cancellation_waits_for_lease_release)'
threads-required = "num-test-threads"
# These real IAM/state fixtures keep production lock-acquisition limits while
# peers complete durable IO. Reserve capacity from unrelated test processes;
# the shared fixture's durable_iam_state_ prefix keeps the whole family together.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::durable_iam_state_/)'
threads-required = "num-test-threads"
# Real target-repair and rebalance-stop fixtures commit durable metadata while
# peers wait on production locks or bounded cancellation checks. Reserve IO
# capacity for each fixture; its internal races and deadlines remain intact.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(/^admin::handlers::(replication::target_repair_tests::|rebalance::rebalance_handler_tests::real_admin_stop_)/)'
threads-required = "num-test-threads"
# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
# Their existing deadlines, assertions, and retry policy remain in force.
[[profile.default.overrides]]
filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
threads-required = "num-test-threads"
# Serialize the durable manual-transition checkpoint test across nextest's
@@ -218,6 +262,13 @@ test-group = 'e2e-cluster-nightly'
filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
test-group = 'e2e-vault'
# These server-heavy smoke cases perform 1k+ sequential S3 writes or run the
# scanner and a remote tier server together. Reserve the full test budget so
# unrelated E2E processes cannot starve their storage and polling deadlines.
[[profile.default.overrides]]
filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
threads-required = "num-test-threads"
# This four-disk, 65-member rollback probe already drives up to 32 concurrent
# durable deletions. Reserve this nextest run's capacity for its progress oracle.
[[profile.default.overrides]]
@@ -257,6 +308,13 @@ setup = 'lifecycle-large-stack'
filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
setup = 'ecstore-large-stack'
# These real-disk scanner fixtures assert quiet-workload publication and
# durable MRF admission. Reserve the pool while their bounded CAS/fsync
# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
[[profile.ci.overrides]]
filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
threads-required = "num-test-threads"
# ===========================================================================
# QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
#
@@ -282,13 +340,6 @@ retries = 2
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
test-group = 'ecstore-serial-flaky'
# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
# on producer/consumer timing windows that stretch past the budget on loaded
# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(walk_dir_does_not_charge_consumer_backpressure_to_the_stall_budget)'
retries = 2
# Serialize the relocated-pool GET resume regression under the ci profile too
# (see the matching default-profile override near the top). No longer a
# quarantine: the fixture race (rustfs#6701/rustfs#6703) was fixed by #6707,
@@ -320,6 +371,11 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
test-group = 'ecstore-serial-flaky'
# Serialize the replacement bucket metadata heal test under the ci profile too.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
test-group = 'ecstore-serial-flaky'
# Serialize the metadata-cache generation-retirement pair under the ci
# profile too (see the matching default-profile override near the top). Not a
# quarantine: no retries.
@@ -334,7 +390,30 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
test-group = 'embedded-test-ports'
[[profile.ci.overrides]]
filter = 'package(rustfs) & binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
threads-required = "num-test-threads"
# Preserve the cancellation probe's startup budget and assertions in CI too.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(=connect::diagnostics::trace_runtime::tests::local_runtime_profile_cancellation_waits_for_lease_release)'
threads-required = "num-test-threads"
# Keep the same IAM/state fixture capacity reservation in CI without changing the
# production lock deadline, internal concurrency, assertions, or retry policy.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::durable_iam_state_/)'
threads-required = "num-test-threads"
# Match the real admin fixture capacity reservation without retries or changes
# to production lock deadlines, cancellation barriers, or assertions.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(/^admin::handlers::(replication::target_repair_tests::|rebalance::rebalance_handler_tests::real_admin_stop_)/)'
threads-required = "num-test-threads"
# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
# Their existing deadlines, assertions, and retry policy remain in force.
[[profile.ci.overrides]]
filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
threads-required = "num-test-threads"
# Serialize the durable manual-transition checkpoint test under the ci profile
@@ -468,6 +547,7 @@ default-filter = """
| test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
| test(/^reliant::lifecycle::/)
| test(/^reliant::tiering::/)
| test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
| test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
)
"""
@@ -476,13 +556,16 @@ fail-fast = false
[profile.e2e-smoke.junit]
path = "junit.xml"
# The pagination boundary cases can stall when a server/listing regression
# prevents the continuation request from completing. Keep the timeout scoped
# to those known failure modes so legitimate lifecycle/tiering waits retain
# their test-level timing budget.
# Bound the whole pagination fixture, including startup, seeding, and cleanup;
# its LIST deadline begins only after the seed objects have been written.
[[profile.e2e-smoke.overrides]]
filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
threads-required = "num-test-threads"
[[profile.e2e-smoke.overrides]]
filter = 'package(e2e_test) & test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry)'
threads-required = "num-test-threads"
# ---------------------------------------------------------------------------
# e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
@@ -692,6 +775,7 @@ default-filter = """
& !test(/^replication_extension_test::/)
& !test(/^replication_target_matrix_test::/)
& !test(/^on_demand_migration::(concurrency_test|fault_test|interop_test|real_source_test)::/)
& !test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
"""
fail-fast = false
@@ -720,3 +804,9 @@ test-group = 'e2e-cluster-nightly'
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
test-group = 'e2e-vault'
# Keep the storage-heavy smoke cases isolated in the full suite too. They
# start multiple server fixtures and issue many sequential S3 operations.
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
threads-required = "num-test-threads"
-38
View File
@@ -1,38 +0,0 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: ''
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
**Smartphone (please complete the following information):**
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
**Additional context**
Add any other context about the problem here.
+110
View File
@@ -0,0 +1,110 @@
name: Bug report
description: Create a report to help us improve
body:
- type: markdown
attributes:
value: |
### Storage support requirements
**Supported:** local disks, SAN (Fibre Channel) attached volumes, or JBOD-mounted disks. We recommend formatting them with the **XFS** file system.
**Not supported:** NFS, SMB/Samba/CIFS, and other remote or shared filesystems as RustFS data directories. This also applies to the underlying host storage for Docker volumes and Kubernetes PersistentVolumes.
Reports that only reproduce on unsupported storage will be closed without investigation. Please reproduce the issue on supported storage before submitting a bug report.
See [Disk Planning and File System Selection](https://docs.rustfs.com/en/installation/linux/prerequisites-and-service) for details.
- type: checkboxes
id: storage-support
attributes:
label: Supported storage confirmation
options:
- label: I have reproduced this issue with all RustFS data directories on supported storage (local disks, SAN/Fibre Channel, or JBOD; XFS recommended), without NFS, SMB/Samba/CIFS, or other remote or shared filesystem mounts, including storage backing Docker volumes and Kubernetes PersistentVolumes.
required: true
- type: textarea
id: bug-description
attributes:
label: Describe the bug
description: A clear and concise description of what happened.
validations:
required: true
- type: textarea
id: reproduction-steps
attributes:
label: Steps to reproduce
description: Include the commands, configuration, or actions needed to reproduce the problem.
placeholder: |
1. Start RustFS with ...
2. Run ...
3. Observe ...
validations:
required: true
- type: textarea
id: expected-behavior
attributes:
label: Expected behavior
description: A clear and concise description of what you expected to happen.
validations:
required: true
- type: input
id: rustfs-version
attributes:
label: RustFS version
description: Provide the release version, container image tag, or commit hash.
validations:
required: true
- type: dropdown
id: deployment-method
attributes:
label: Deployment method
description: How are you running RustFS? Add any relevant details in Additional context.
options:
- Binary
- Docker / Docker Compose
- Kubernetes / Helm
- Built from source
- Other
- type: textarea
id: storage-mounts
attributes:
label: Data directory filesystems and mounts
description: |
Identify the filesystem type and mount source for every RustFS data directory.
On Linux, include output from `findmnt -T /path/to/data -o TARGET,SOURCE,FSTYPE,OPTIONS` for each data path; on other systems, provide equivalent mount information.
For Docker, identify the backing host path and filesystem, not just the container path or volume name.
For Kubernetes, identify the PV, StorageClass/CSI driver, and actual backing filesystem; a PVC name alone is not sufficient.
Remove credentials and other sensitive information before submitting.
placeholder: |
RustFS data paths:
Filesystem types and mount sources:
Mount command output:
Docker host paths or Kubernetes volume backing storage (if applicable):
validations:
required: true
- type: input
id: operating-system
attributes:
label: Server operating system and architecture
description: Include the operating system version and CPU architecture where RustFS is running.
placeholder: "For example: Ubuntu 24.04, x86_64"
validations:
required: true
- type: textarea
id: client-environment
attributes:
label: Client environment
description: Include the S3 client or SDK and version. For console issues, include the browser, operating system, and desktop or mobile device details.
- type: textarea
id: logs
attributes:
label: Relevant logs
description: Paste relevant log output. Remove credentials and other sensitive information before submitting.
render: shell
- type: textarea
id: screenshots
attributes:
label: Screenshots
description: If applicable, drag and drop screenshots here to help explain the problem.
- type: textarea
id: additional-context
attributes:
label: Additional context
description: Add any other context, such as deployment topology or relevant configuration. Remove credentials and other sensitive information before submitting.
+1
View File
@@ -0,0 +1 @@
blank_issues_enabled: false
-20
View File
@@ -1,20 +0,0 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
@@ -0,0 +1,38 @@
name: Feature request
description: Suggest an idea for this project
body:
- type: checkboxes
id: request-scope
attributes:
label: Request scope
description: |
RustFS data directories backed by NFS, SMB/Samba/CIFS, or other remote or shared filesystems are unsupported, including those behind Docker volumes or Kubernetes PersistentVolumes.
Troubleshooting requests for these deployments will be closed without investigation. Use this form for new or improved functionality.
options:
- label: This is a feature request and does not ask for troubleshooting an unsupported remote or shared filesystem deployment.
required: true
- type: textarea
id: problem-description
attributes:
label: Problem or use case
description: Describe the problem or use case that motivates this feature request.
placeholder: "For example: I am trying to ... but ..."
validations:
required: true
- type: textarea
id: proposed-solution
attributes:
label: Proposed solution
description: A clear and concise description of what you would like to happen.
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
description: Describe any alternative solutions or features you have considered.
- type: textarea
id: additional-context
attributes:
label: Additional context
description: Add any other context or drag and drop screenshots about the feature request here.
+2 -7
View File
@@ -19,7 +19,7 @@ runs:
using: composite
steps:
- name: Install quality tools
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
with:
tool: |
ripgrep@15.2.0
@@ -100,12 +100,7 @@ runs:
- name: Check test wiring
shell: bash
run: |
python3 ./scripts/check_test_wiring.py --self-test
python3 ./scripts/check_scheduled_validation_freshness.py --self-test
python3 ./scripts/test_security_workflow.py
python3 ./scripts/test_nightly_candidate.py
python3 ./scripts/check_test_wiring.py
run: python3 ./scripts/check_test_wiring.py
- name: Check no planning docs committed
shell: bash
+3 -1
View File
@@ -107,9 +107,11 @@ runs:
- name: Install cargo-nextest
if: inputs.install-test-tools == 'true'
uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
with:
tool: cargo-nextest@0.9.138
- name: Setup Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
with:
# false is rust-cache's own default. With true, cleanup.ts returns
# *before* pruning ~/.cargo/registry/src, and config.ts archives the
@@ -1,61 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Architecture Migration Rules
on:
pull_request:
types: [ opened, synchronize, reopened, closed ]
branches: [ main ]
paths:
- "ARCHITECTURE.md"
- "docs/architecture/**"
- "scripts/check_architecture_migration_rules.sh"
- ".github/workflows/architecture-migration-rules.yml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
architecture-migration-rules:
name: Architecture Migration Rules
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install ripgrep
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
with:
tool: ripgrep@15.2.0
- name: Check architecture migration rules
run: ./scripts/check_architecture_migration_rules.sh
+19 -10
View File
@@ -24,8 +24,11 @@ on:
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
- 'scripts/security/check_performance_ab_workflow.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_tier_artifact_workflow.sh'
@@ -40,8 +43,11 @@ on:
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
- 'scripts/security/check_performance_ab_workflow.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_tier_artifact_workflow.sh'
@@ -71,7 +77,7 @@ jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Explain cancellation run
@@ -83,11 +89,11 @@ jobs:
cargo-deny:
name: Cargo Deny
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 20
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -106,7 +112,7 @@ jobs:
# default is now "false", but state it explicitly — see
# scripts/security/check_cache_save_if.sh.
- name: Setup Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
with:
# Same reasoning as the setup composite: true archives every
# dependency's unpacked source tree.
@@ -116,7 +122,7 @@ jobs:
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install cargo-deny
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
with:
tool: cargo-deny
@@ -126,11 +132,11 @@ jobs:
workflow-pin-report:
name: Workflow Pin Report
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 5
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -158,9 +164,12 @@ jobs:
- name: Check package version contract
run: ./scripts/test_package_versions.sh
- name: Check package service lifecycle contract
run: ./scripts/test_package_service_scripts.sh
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
if: github.event_name == 'pull_request' && github.event.action != 'closed'
permissions:
@@ -168,7 +177,7 @@ jobs:
pull-requests: write
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -201,7 +210,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+120 -87
View File
@@ -27,7 +27,8 @@
# schedule and manual dispatch build the full platform matrix.
#
# Manual Parameters:
# - build_docker: Build and push Docker images (default: true)
# - publish: Publish binaries/releases for manual runs (default: false)
# - build_docker: Advisory only; manual runs do not trigger Docker publishing
# - platforms: Comma-separated platform IDs or 'all' (default: all)
name: Build and Release
@@ -55,12 +56,16 @@ on:
- cron: "13 1 * * 0" # Weekly on Sunday 01:13 UTC
workflow_dispatch:
inputs:
publish:
description: "Publish to download channels and GitHub Releases (false keeps only workflow artifacts)"
required: false
default: false
type: boolean
build_docker:
# Advisory only. docker.yml triggers on workflow_run and its job-level
# condition requires the triggering event to be a tag push, so a manual
# dispatch of this workflow never produces images regardless of this
# value. Kept because the summary step reports it; wiring it up would
# mean teaching docker.yml's version parser a second event shape.
# value. Publishing remains controlled by the explicit publish input.
description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
required: false
default: true
@@ -88,17 +93,18 @@ jobs:
# Build strategy check - determine build type based on trigger
build-check:
name: Build Strategy Check
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_publish: ${{ github.event_name != 'workflow_dispatch' || inputs.publish == true }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -172,7 +178,7 @@ jobs:
prepare-platform-matrix:
name: Prepare Platform Matrix
needs: build-check
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
outputs:
matrix: ${{ steps.select.outputs.matrix }}
@@ -208,9 +214,9 @@ jobs:
all='{"include":[
{"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
{"target_id":"linux-aarch64-musl","os":"sm-standard-4","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":"","features":""},
{"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
{"target_id":"linux-aarch64-gnu","os":"sm-standard-4","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope"},
{"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""}
]}'
@@ -257,7 +263,7 @@ jobs:
matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0
@@ -402,6 +408,8 @@ jobs:
- name: Build RustFS
shell: bash
env:
BUILD_TARGET: ${{ matrix.target }}
run: |
# Force rebuild by touching build.rs
touch rustfs/build.rs
@@ -418,14 +426,37 @@ jobs:
# Build one binary at a time so release LTO links cannot exhaust a
# self-hosted runner by running concurrently.
for binary in "${BINARY_NAMES[@]}"; do
PROFILE_ARGS=()
if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
# The in-process profiler resolves names from the final executable.
# Retain function symbols without enabling DWARF source information.
PROFILE_ARGS=(--config 'profile.release.strip="none"')
fi
if [[ "${{ matrix.cross }}" == "true" ]]; then
# All cross targets in the matrix are Linux; zigbuild handles them.
cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" -p rustfs --bin "$binary"
cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
else
cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" -p rustfs --bin "$binary"
cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
fi
done
- name: Generate the CPU symbol catalog
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
shell: bash
run: |
cargo metadata --locked --format-version 1 --filter-platform '${{ matrix.target }}' \
--features rustfs/pyroscope > "$RUNNER_TEMP/cpu-catalog-metadata.json"
features=$(jq -ce '
(.packages[] | select(.name == "rustfs") | .id) as $id |
.resolve.nodes[] | select(.id == $id) | .features | map(select(. != "default")) | sort
' "$RUNNER_TEMP/cpu-catalog-metadata.json")
release_dir="target/${{ matrix.target }}/release"
# These task-owned outputs may remain in a restored target cache.
rm -f "$release_dir/rustfs.cpu-symbol-catalog.json" "$release_dir/rustfs.cpu-symbol-catalog.sha256"
cargo run --locked -p rustfs-connect-symbol-catalog -- \
"$release_dir/rustfs" "$(git rev-parse HEAD)" '${{ matrix.target }}' "$features" \
"$release_dir/rustfs.cpu-symbol-catalog.json" > "$release_dir/rustfs.cpu-symbol-catalog.sha256"
- name: Create release package
id: package
shell: bash
@@ -516,6 +547,10 @@ jobs:
BINARY_FILES+=("rustfs-cli")
fi
if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256)
fi
# Verify the binaries exist before packaging
for binary in "${BINARY_FILES[@]}"; do
if [[ ! -f "$binary" ]]; then
@@ -625,7 +660,7 @@ jobs:
echo "🔧 Build type: ${BUILD_TYPE}"
echo "📊 Version: ${VERSION}"
- name: Verify packaged Linux CLI
- name: Verify packaged Linux binaries
if: matrix.platform == 'linux'
shell: bash
run: |
@@ -634,21 +669,27 @@ jobs:
package_file="${{ steps.package.outputs.package_file }}"
archive_members="$(unzip -Z1 "$package_file" | sort)"
expected_members=$'rustfs\nrustfs-cli'
target="${{ matrix.target }}"
features="${{ matrix.features }}"
if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then
expected_members+=$'\nrustfs.cpu-symbol-catalog.json\nrustfs.cpu-symbol-catalog.sha256'
fi
if [[ "$archive_members" != "$expected_members" ]]; then
echo "Linux package must contain exactly rustfs and rustfs-cli" >&2
echo "Linux package does not contain the expected files" >&2
printf '%s\n' "$archive_members" >&2
exit 1
fi
if [[ "${{ matrix.target }}" == "x86_64-unknown-linux-gnu" ]]; then
cli_dir="$(mktemp -d)"
trap 'rm -rf "$cli_dir"' EXIT
unzip -q "$package_file" rustfs-cli -d "$cli_dir"
"$cli_dir/rustfs-cli" --help >/dev/null
fi
# Every Linux target builds natively, so executing the binaries here
# proves each artifact matches its runner architecture.
cli_dir="$(mktemp -d)"
trap 'rm -rf "$cli_dir"' EXIT
unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir"
"$cli_dir/rustfs" --version
"$cli_dir/rustfs-cli" --help >/dev/null
- name: Verify packaged console
if: matrix.target == 'x86_64-unknown-linux-gnu'
if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
shell: bash
run: |
set -euo pipefail
@@ -702,14 +743,14 @@ jobs:
exit 1
- name: Upload to GitHub artifacts
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ steps.package.outputs.package_name }}
path: "rustfs-*.zip"
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}
- name: Upload to Cloudflare R2
if: env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
if: needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
@@ -726,12 +767,33 @@ jobs:
fi
# The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted
# images did). Install it on demand so R2 uploads survive a fresh runner
# instead of hard-failing here.
# images did). Install the official CLI bundle on demand so R2 uploads
# survive a fresh runner without relying on a distro package.
if ! command -v aws >/dev/null 2>&1; then
echo "aws CLI not found on runner; installing..."
if command -v apt-get >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
aws_arch="$(uname -m)"
case "$aws_arch" in
x86_64) aws_arch="x86_64" ;;
aarch64|arm64) aws_arch="aarch64" ;;
*)
echo "❌ Unsupported Linux architecture for AWS CLI: $aws_arch"
exit 1
;;
esac
aws_install_root="${RUNNER_TEMP:-$PWD}/aws-cli"
rm -rf "$aws_install_root"
mkdir -p "$aws_install_root"
curl --fail --location --retry 3 \
"https://awscli.amazonaws.com/awscli-exe-linux-${aws_arch}.zip" \
--output "$aws_install_root/awscliv2.zip"
unzip -q "$aws_install_root/awscliv2.zip" -d "$aws_install_root"
"$aws_install_root/aws/install" \
--install-dir "$aws_install_root/aws-cli" \
--bin-dir "$aws_install_root/bin"
export PATH="$aws_install_root/bin:$PATH"
aws --version
elif command -v brew >/dev/null 2>&1; then
brew install awscli
else
@@ -770,70 +832,41 @@ jobs:
name: Build Summary
needs: [ build-check, build-rustfs ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Build completion summary
shell: bash
env:
# dispatch input via env: free-form text must not be pasted into the
# script for bash to evaluate.
INPUT_BUILD_DOCKER: ${{ github.event.inputs.build_docker }}
SHOULD_PUBLISH: ${{ needs.build-check.outputs.should_publish }}
BUILD_TYPE: ${{ needs.build-check.outputs.build_type }}
VERSION: ${{ needs.build-check.outputs.version }}
BUILD_STATUS: ${{ needs.build-rustfs.result }}
EVENT_NAME: ${{ github.event_name }}
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
echo "🎉 Build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo ""
# Check build status
BUILD_STATUS="${{ needs.build-rustfs.result }}"
echo "📊 Build Results:"
echo " 📦 All platforms: $BUILD_STATUS"
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview artifacts are published in a GitHub prerelease"
echo "⏭️ Preview releases do not update latest channels"
;;
"development")
echo "🛠️ Development build artifacts have been uploaded to OSS dev directory"
echo "⚠️ This is a development build - not suitable for production use"
;;
"release")
echo "🚀 Release build artifacts have been uploaded to OSS release directory"
echo "✅ This build is ready for production use"
echo "🏷️ GitHub Release will be created in this workflow"
;;
"prerelease")
echo "🧪 Prerelease build artifacts have been uploaded to OSS release directory"
echo "⚠️ This is a prerelease build - use with caution"
echo "🏷️ GitHub Release will be created in this workflow"
;;
esac
echo ""
echo "🐳 Docker Images:"
if [[ "$BUILD_TYPE" == "preview" ]]; then
echo "⏭️ Preview tags do not publish Docker images"
elif [[ "$INPUT_BUILD_DOCKER" == "false" ]]; then
echo "⏭️ Docker image build was skipped (binary only build)"
echo "Build type: $BUILD_TYPE"
echo "Version: $VERSION"
echo "Build result: $BUILD_STATUS"
if [[ "$SHOULD_PUBLISH" != "true" ]]; then
echo "Artifact-only run: no download channels or GitHub Releases are updated."
echo "Successful platform packages are available as workflow artifacts."
elif [[ "$BUILD_STATUS" == "success" ]]; then
echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
echo "Publishing enabled; check the upload and release jobs for their results."
else
echo "❌ Docker image build will be skipped due to build failure"
echo "Build incomplete; check individual platform jobs for any completed uploads."
fi
if [[ "$EVENT_NAME" == "push" && "$GITHUB_REF" == refs/tags/* && "$BUILD_STATUS" == "success" ]]; then
echo "Docker publishing is eligible through the downstream workflow."
else
echo "This run does not trigger Docker publishing."
fi
# Create GitHub Release for every valid release tag, including previews
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
@@ -842,7 +875,7 @@ jobs:
release_url: ${{ steps.create.outputs.release_url }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0
@@ -893,20 +926,20 @@ jobs:
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
@@ -987,8 +1020,8 @@ jobs:
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: sm-standard-2
timeout-minutes: 30
steps:
- name: Update latest.json
@@ -1047,8 +1080,8 @@ jobs:
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
@@ -1085,8 +1118,8 @@ jobs:
cleanup-preview-releases:
name: Cleanup Preview Releases
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
@@ -1140,7 +1173,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+10 -10
View File
@@ -15,16 +15,16 @@
# Sole writer of the Rust dependency caches that ci.yml restores.
#
# Why this is a separate workflow rather than steps inside ci.yml: ci.yml's
# concurrency group cancels in-progress runs on main pushes, and merges land far
# faster than its 70-minute pipeline. Measured over 15 consecutive main pushes:
# concurrency group originally cancelled in-progress main runs, while merges
# landed faster than its 70-minute pipeline. Over 15 consecutive main pushes:
# 12 cancelled, 2 failed, 0 succeeded. A cancelled run never reaches
# Swatinem/rust-cache's post step (cache-on-failure does not cover cancellation),
# so the writer lanes were saving nothing and every PR paid a cold restore —
# 11.8-20.9 minutes of "Setup Rust environment" against 0.7-3.4 warm.
#
# Splitting cache writing out of the test pipeline lets ci.yml keep cancelling
# superseded runs (which is correct — nobody needs test results for a commit
# that is already three merges behind) while the caches still get written.
# Keep cache writing separate from validation: only these jobs publish the
# complete feature closure for each key. Main validation now also finishes its
# running baseline, while superseded PR attempts can still be cancelled.
#
# The group below deliberately does NOT cancel in progress; see the comment on
# it for how that bounds concurrency and why it is scoped by event.
@@ -109,7 +109,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -141,7 +141,7 @@ jobs:
- name: Upload cargo timings report
if: inputs.emit_timings
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: cargo-timings-ci-dev
path: target/cargo-timings/
@@ -199,7 +199,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -230,7 +230,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -260,7 +260,7 @@ jobs:
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
+133 -91
View File
@@ -47,12 +47,12 @@ permissions:
contents: read
# Concurrency groups are scoped per event so different triggers never cancel
# each other: PR pushes cancel the previous run of that PR, main pushes keep
# latest-wins semantics among themselves, and scheduled runs always complete
# (a shared group used to let every merge kill the weekly scheduled run).
# each other. PR pushes cancel superseded attempts; main and scheduled runs
# finish so a busy merge stream cannot starve the complete baseline. For main,
# GitHub retains one running run and replaces the pending run with the latest.
concurrency:
group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
CARGO_TERM_COLOR: always
@@ -63,7 +63,7 @@ jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Explain cancellation run
@@ -72,12 +72,12 @@ jobs:
classify-changes:
name: Select CI scope
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
outputs:
mode: ${{ steps.scope.outputs.mode }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 2
persist-credentials: false
@@ -98,24 +98,28 @@ jobs:
typos:
name: Typos
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install typos download dependency
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends wget
- name: Typos check with custom config file
uses: crate-ci/typos@37bb98842b0d8c4ffebdb75301a13db0267cef89 # master
uses: crate-ci/typos@512fc24f32f44ab01972217aaaf3dc86ec234d53 # v1.50.2
# Fail early with compile-free checks for every pull request.
quick-checks:
name: Quick Checks
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -124,7 +128,7 @@ jobs:
test-and-lint:
name: Workspace Test and Lint
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
@@ -132,7 +136,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# Checkout otherwise writes the token into .git/config, where a PR's
# own build.rs or proc-macro could read it back out.
@@ -142,11 +146,9 @@ jobs:
uses: ./.github/actions/setup
with:
rust-version: stable
# Every lane in this workflow reads its cache and none writes it.
# cache-warm.yml is the sole writer for all four keys: this workflow
# cancels superseded runs on main, and a cancelled run never reaches
# rust-cache's post step, so writing from here saved nothing (12 of 15
# consecutive main-push runs were cancelled). See rustfs/backlog#1600.
# Every lane reads its cache; cache-warm.yml remains the sole writer
# with the matching feature closure. Tests never compete to replace
# shared caches, including when a PR attempt is cancelled.
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
@@ -191,13 +193,10 @@ jobs:
# Raised to 3 on main pushes and manual dispatches; PRs keep 2 so the
# merge path is untouched while the experiment runs.
#
# Dispatch is included because push alone cannot supply the samples:
# this workflow cancels superseded runs on main, and only 4 of the last
# 20 push-triggered Test and Lint jobs reached a terminal state — at
# that rate ten samples would take roughly fifty merges. The
# concurrency group is scoped by event_name, so a dispatched run has
# its own group and is not cancelled by merge traffic, which makes the
# sample collectable on demand rather than by waiting.
# During baseline collection, main pushes cancelled superseded runs:
# only 4 of 20 Test and Lint jobs reached a terminal state. Running
# main validations now finish; dispatch remains an independent group
# for collecting additional samples on demand.
#
# Baseline over 17 samples at 2:
# median nextest/clippy step ratio 1.95, spread 1.85-2.06. The gate-2
@@ -246,11 +245,11 @@ jobs:
mkdir -p artifacts/test-and-lint
set +e
timeout --verbose --signal=TERM --kill-after=30s 15m \
cargo test --all --doc \
cargo test --all --exclude e2e_test --doc \
2>&1 | tee artifacts/test-and-lint/doctest.log
status=${PIPESTATUS[0]}
{
echo "command=cargo test --all --doc"
echo "command=cargo test --all --exclude e2e_test --doc"
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
@@ -261,7 +260,7 @@ jobs:
- name: Upload test reports and diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: junit-test-and-lint-${{ github.run_number }}
path: |
@@ -277,21 +276,10 @@ jobs:
- name: Check log-analyzer rule anchors
run: ./scripts/check_log_analyzer_rules.sh
# Explicit gate for migration-critical suites. These tests already ran in
# the full nextest pass above; a single filtered nextest invocation keeps
# the named gate without rebuilding or re-running them one package at a time.
#
# The gate selects tests by name substring (data_movement / rebalance /
# decommission / source_cleanup / delete_marker), so renames can silently
# thin it. The script owns the filter expression and first verifies the
# selected-test count against the committed floor in
# .config/migration-gate-floor.txt before running the gate; renames or
# removals must update that file consciously (see the script header).
# Kept on the default profile (no --profile ci): a second --profile ci run
# would clobber target/nextest/ci/junit.xml, and none of these tests are
# quarantined so they gain nothing from the ci profile's retry overrides.
- name: Run rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh
# Preserve the migration floor and require successful execution evidence
# from the workspace run, without compiling or executing its subset again.
- name: Verify rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
# This gate builds into a fresh target directory to isolate the E2E root.
# Give its cold build a separate budget from workspace tests and migration proofs.
@@ -305,7 +293,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -344,7 +332,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -391,7 +379,7 @@ jobs:
- name: Upload ILM test diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
path: |
@@ -408,7 +396,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -427,6 +415,7 @@ jobs:
run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
- name: Run rio-v2 feature tests
shell: bash
env:
# Match the main nextest lane's #5394 link-I/O guard. A cold feature
# cache otherwise fans out enough rust-lld processes to exhaust this
@@ -436,9 +425,25 @@ jobs:
# --profile ci so the quarantine list (and its junit flaky markers)
# covers this leg too; the default profile is the local no-retry
# profile and silently ignored quarantined flakes here (rustfs#6703).
cargo nextest run --profile ci -p rustfs -p rustfs-ecstore --features rio-v2
mkdir -p artifacts/test-and-lint
rm -f target/nextest/ci/junit.xml
./scripts/ci/resource_sampler.sh start nextest-rio-v2
trap './scripts/ci/resource_sampler.sh stop' EXIT
cargo nextest run --profile ci -p rustfs -p rustfs-ecstore --features rio-v2 \
2>&1 | tee artifacts/test-and-lint/nextest.log
cargo test -p rustfs --doc --features rio-v2
- name: Upload rio-v2 test reports and diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: junit-test-and-lint-rio-v2-${{ github.run_number }}
path: |
target/nextest/ci/junit.xml
artifacts/test-and-lint
retention-days: 3
if-no-files-found: error
connect-short-credential-boundary:
name: Connect Short Credential Boundary
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
@@ -449,7 +454,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -504,13 +509,9 @@ jobs:
runs-on: sm-standard-4
timeout-minutes: 90
strategy:
# On a PR, one failing protocol leg is enough to know the PR is not ready,
# so stop the sibling leg instead of paying another ~40 minutes for it.
# Everywhere else (main pushes, the merge queue, the weekly schedule) keep
# the full signal: there we want to know whether swift AND sftp are broken,
# not just whichever failed first. This is the only part of the early-stop
# work that also covers fork PRs, since it needs no token.
fail-fast: ${{ github.event_name == 'pull_request' }}
# Preserve both independent results when one protocol fails, so diagnosis
# and a focused fix do not require rebuilding an interrupted sibling.
fail-fast: false
matrix:
features:
- name: swift
@@ -521,7 +522,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -533,27 +534,50 @@ jobs:
cache-save-if: 'false'
install-build-packaging-tools: 'false'
- name: Protect Connect test home
run: chmod go-w "$(realpath "$HOME")"
- name: Prepare protocol test state
run: |
rm -f target/nextest/ci/junit.xml
chmod go-w "$(realpath "$HOME")"
- name: Run clippy with ${{ matrix.features.name }}
run: |
./scripts/ci/resource_sampler.sh start clippy
trap './scripts/ci/resource_sampler.sh stop' EXIT
cargo clippy -p rustfs -p rustfs-protocols --all-targets ${{ matrix.features.flags }} -- -D warnings
- name: Run tests with ${{ matrix.features.name }}
id: protocol-tests
shell: bash
env:
# Keep feature-test linking under the same bounded concurrency as the
# main nextest lane; Clippy is metadata-only and needs no such limit.
CARGO_BUILD_JOBS: "2"
SAMPLER_INTERVAL_SECS: "15"
run: |
# --profile ci so the quarantine list (and its junit flaky markers)
# covers this leg too; the default profile is the local no-retry
# profile and silently ignored quarantined flakes here (rustfs#6703).
cargo nextest run --profile ci -p rustfs -p rustfs-protocols ${{ matrix.features.flags }}
mkdir -p artifacts/protocol-tests
./scripts/ci/resource_sampler.sh start nextest
trap './scripts/ci/resource_sampler.sh stop' EXIT
cargo nextest run --profile ci -p rustfs -p rustfs-protocols ${{ matrix.features.flags }} \
2>&1 | tee artifacts/protocol-tests/nextest.log
- name: Upload protocol test reports and diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: junit-test-and-lint-${{ matrix.features.name }}-${{ github.run_number }}-${{ github.run_attempt }}
path: |
target/nextest/ci/junit.xml
artifacts/protocol-tests
artifacts/test-and-lint
retention-days: 3
if-no-files-found: warn
build-rustfs-debug-binary:
name: Build RustFS Debug Binary
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 30
@@ -561,7 +585,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -621,7 +645,7 @@ jobs:
PYBUILD
- name: Upload debug binary
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-debug-binary
path: |
@@ -646,7 +670,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -662,7 +686,7 @@ jobs:
run: python3 scripts/e2e_binary.py build --bins --features rio-v2,e2e-test-hooks
- name: Upload debug binary
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-debug-binary-rio-v2
path: |
@@ -690,7 +714,7 @@ jobs:
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -754,7 +778,7 @@ jobs:
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -770,7 +794,7 @@ jobs:
install-build-packaging-tools: 'false'
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -785,7 +809,7 @@ jobs:
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
@@ -806,6 +830,10 @@ jobs:
python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
# Keep E2E documentation coverage with its already-built package graph.
- name: Run E2E documentation tests
run: cargo test -p e2e_test --doc
# PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
# profile.e2e-smoke default-filter in .config/nextest.toml is the single
# wiring mechanism for e2e tests in CI — extend that filter instead of
@@ -815,23 +843,27 @@ jobs:
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
SAMPLER_INTERVAL_SECS: "15"
run: |
./scripts/ci/resource_sampler.sh start e2e-smoke
trap './scripts/ci/resource_sampler.sh stop' EXIT
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
--status-level all --final-status-level all --failure-output final
- name: Upload e2e smoke diagnostics
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-diagnostics-${{ github.run_number }}
path: |
${{ runner.temp }}/rustfs-e2e-smoke-logs/
${{ runner.temp }}/rustfs-e2e-smoke-list.json
artifacts/test-and-lint
if-no-files-found: warn
- name: Upload e2e smoke JUnit report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-junit-${{ github.run_number }}
path: target/nextest/e2e-smoke/junit.xml
@@ -856,7 +888,7 @@ jobs:
- name: Upload test logs
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-test-logs-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
@@ -879,7 +911,7 @@ jobs:
timeout-minutes: 55
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -903,7 +935,7 @@ jobs:
fi
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -945,7 +977,7 @@ jobs:
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
@@ -1001,11 +1033,12 @@ jobs:
RUSTFS_E2E_STARTUP_CAS_BINARY: ${{ runner.temp }}/rustfs-startup-cas-input/rustfs
RUSTFS_E2E_STARTUP_CAS_BUILD_MANIFEST: ${{ runner.temp }}/rustfs-startup-cas-input/rustfs.e2e-startup-cas-build.json
RUSTFS_E2E_STARTUP_CAS_ARTIFACT_DIR: ${{ runner.temp }}/rustfs-startup-cas-evidence
RUSTFS_HEAL_CHAOS_LOG_ROOT: ${{ runner.temp }}/rustfs-heal-chaos-logs
run: python3 scripts/e2e_binary.py run --binary "$RUSTFS_E2E_STARTUP_CAS_BINARY" --features e2e-test-hooks -- cargo nextest run --profile e2e-full -p e2e_test
- name: Upload junit
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-full-junit-${{ github.run_number }}
path: |
@@ -1015,7 +1048,7 @@ jobs:
- name: Upload startup CAS evidence
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: fresh-startup-cas-evidence-${{ github.run_number }}
path: |
@@ -1024,6 +1057,15 @@ jobs:
if-no-files-found: warn
retention-days: 7
- name: Upload heal cluster logs
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-full-heal-logs-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-heal-chaos-logs
if-no-files-found: warn
retention-days: 7
e2e-tests-rio-v2:
name: End-to-End Tests (rio-v2)
# Inherits the schedule/dispatch-only gate through needs: on every other
@@ -1034,7 +1076,7 @@ jobs:
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -1044,7 +1086,7 @@ jobs:
rm -f /tmp/rustfs.log
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary-rio-v2
path: target/debug
@@ -1077,7 +1119,7 @@ jobs:
- name: Upload test logs
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-test-logs-rio-v2-${{ github.run_number }}
path: /tmp/rustfs.log
@@ -1090,12 +1132,12 @@ jobs:
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
@@ -1120,7 +1162,7 @@ jobs:
- name: Upload s3 test artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: s3tests-implemented-${{ github.run_number }}
path: artifacts/s3tests-single/**
@@ -1164,12 +1206,12 @@ jobs:
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
@@ -1202,7 +1244,7 @@ jobs:
- name: Upload s3 test artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: s3tests-lifecycle-behavior-${{ github.run_number }}
path: artifacts/s3tests-single/**
@@ -1230,10 +1272,10 @@ jobs:
- build-rustfs-debug-binary-rio-v2
- e2e-tests-rio-v2
- e2e-full
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Require the expected result of every CI lane
@@ -1272,7 +1314,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+3 -3
View File
@@ -46,7 +46,7 @@ jobs:
# Echoes one line; the run exists only so the concurrency group cancels the
# in-flight run of a closed PR.
permissions: {}
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Explain cancellation run
@@ -61,12 +61,12 @@ jobs:
checks: write
issues: write
pull-requests: write
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
steps:
- name: Report CLA result for merge queue
if: github.event_name == 'merge_group'
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
await github.rest.checks.create({
@@ -0,0 +1,198 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Connect diagnostic scheduler receipt acceptance
on:
workflow_dispatch:
inputs:
build_run_id:
description: Main-branch Build and Release run ID
required: true
type: string
artifact_id:
description: Linux x86_64 GNU artifact from that run
required: true
type: string
source_sha:
description: Exact RustFS source commit
required: true
type: string
artifact_digest:
description: GitHub artifact digest including sha256 prefix
required: true
type: string
binary_sha256:
description: Independently verified rustfs binary SHA-256
required: true
type: string
connect_sha:
description: Exact Connect scheduler acceptance harness commit
required: true
type: string
permissions:
actions: read
contents: read
jobs:
diagnostic-schedule:
name: Verify production scheduler receipts over mTLS
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/connect
path: connect-harness
persist-credentials: false
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Verify official source and artifact identity
shell: bash
env:
GH_TOKEN: ${{ github.token }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
SOURCE_SHA: ${{ inputs.source_sha }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
run: |
set -euo pipefail
[[ "$BUILD_RUN_ID" =~ ^[0-9]+$ && "$ARTIFACT_ID" =~ ^[0-9]+$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
[[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
jq -e --arg source "$SOURCE_SHA" --arg repository "$GITHUB_REPOSITORY" '
.head_sha == $source and .head_branch == "main"
and .head_repository.full_name == $repository
and .name == "Build and Release" and .path == ".github/workflows/build.yml"
and .status == "completed" and .conclusion == "success"
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
.workflow_run.id == $run and .workflow_run.head_sha == $source
and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
and .digest == $digest and .expired == false
and (.expires_at | fromdateiso8601) > now
and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
' <<<"$artifact" >/dev/null
gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
printf '%s official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
- name: Extract the verified official binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
run: |
set -euo pipefail
python3 - <<'PY'
import io
import os
import pathlib
import stat
import zipfile
def regular(entry):
mode = entry.external_attr >> 16
return not entry.is_dir() and (stat.S_IFMT(mode) in (0, stat.S_IFREG))
package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
with zipfile.ZipFile('official-artifact.zip') as outer:
entries = outer.infolist()
assert 1 <= len(entries) <= 16
assert len({item.filename for item in entries}) == len(entries)
assert all(regular(item) and pathlib.PurePosixPath(item.filename).name == item.filename for item in entries)
assert sum(item.file_size for item in entries) <= 2147483648
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
assert {item.filename for item in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
assert all(regular(item) and 0 < item.file_size <= 1073741824 for item in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
import shutil
shutil.copyfileobj(source, destination)
PY
chmod 0755 binary/rustfs
printf '%s binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
file binary/rustfs | grep -Eq 'ELF 64-bit LSB.*x86-64'
- name: Run the real scheduler and receipt delivery scenario
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
git -C connect-harness diff --exit-code
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
[[ $(git -C connect-harness diff --name-only) == tests/e2e/connected/rustfs-ref ]]
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/diagnostic-schedule-evidence.test.sh)
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_DIAGNOSTIC_SCHEDULE_EVIDENCE="$GITHUB_WORKSPACE/diagnostic-schedule-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=diagnostic-schedule
- name: Bind and validate sanitized evidence
shell: bash
env:
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
jq -e --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" '
.provenance.connectSha == $connect and .provenance.sourceSha == $source
and .provenance.binarySha256 == $binary
' diagnostic-schedule-evidence.json >/dev/null
jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
--arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" '
.provenance += {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest}
' diagnostic-schedule-evidence.json > diagnostic-schedule-evidence.bound.json
mv diagnostic-schedule-evidence.bound.json diagnostic-schedule-evidence.json
node connect-harness/tests/connectivity/verify-diagnostic-schedule-evidence.mjs diagnostic-schedule-evidence.json --bound
- name: Upload verified receipt evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: connect-diagnostic-schedule-evidence-${{ github.run_id }}
path: diagnostic-schedule-evidence.json
retention-days: 14
if-no-files-found: error
@@ -0,0 +1,236 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Connect bounded health service acceptance
on:
workflow_dispatch:
inputs:
build_run_id:
description: Successful main-branch Build and Release run ID
required: true
type: string
artifact_id:
description: Linux x86_64 GNU artifact from that run
required: true
type: string
source_sha:
description: Exact RustFS source commit
required: true
type: string
artifact_digest:
description: GitHub artifact digest including sha256 prefix
required: true
type: string
binary_sha256:
description: Independently verified rustfs binary SHA-256
required: true
type: string
connect_sha:
description: Exact Connect health service acceptance harness commit
required: true
type: string
permissions:
actions: read
contents: read
jobs:
health-service:
name: Verify bounded health service over mTLS
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/connect
path: connect-harness
persist-credentials: false
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Verify official source and artifact identity
shell: bash
env:
GH_TOKEN: ${{ github.token }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
SOURCE_SHA: ${{ inputs.source_sha }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
run: |
set -euo pipefail
[[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
[[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
jq -e --arg source "$SOURCE_SHA" '
.head_sha == $source and .head_branch == "main"
and .head_repository.full_name == "rustfs/rustfs"
and .name == "Build and Release" and .path == ".github/workflows/build.yml"
and .status == "completed" and .conclusion == "success"
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
.workflow_run.id == $run and .workflow_run.head_sha == $source
and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
and .digest == $digest and .expired == false
and (.expires_at | fromdateiso8601) > now
and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
' <<<"$artifact" >/dev/null
jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
[[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
printf '%s official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
- name: Extract only the verified official service binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
run: |
set -euo pipefail
python3 - <<'PY'
import io
import os
import pathlib
import shutil
import stat
import zipfile
def regular(entry):
mode = entry.external_attr >> 16
return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
with zipfile.ZipFile('official-artifact.zip') as outer:
entries = outer.infolist()
assert 1 <= len(entries) <= 16
assert len({entry.filename for entry in entries}) == len(entries)
assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
assert sum(entry.file_size for entry in entries) <= 2147483648
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
header = source.read(20)
assert header[:7] == b'\x7fELF\x02\x01\x01' and header[18:20] == b'\x3e\x00', 'Expected a little-endian ELF64 x86-64 binary'
destination.write(header)
shutil.copyfileobj(source, destination)
PY
chmod 0755 binary/rustfs
printf '%s binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
- name: Run the real health service job
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
git -C connect-harness diff --exit-code
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
changed=$(git -C connect-harness diff --name-only)
[[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh)
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=health-service
- name: Bind and validate sanitized evidence
shell: bash
env:
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
jq -e --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" '
.provenance.connectSha == $connect and .provenance.sourceSha == $source
and .provenance.binarySha256 == $binary
' health-service-evidence.json >/dev/null
jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
--arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
--arg expiry "$(cat artifact-expires-at)" '
.provenance += {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry}
' health-service-evidence.json > health-service-evidence.bound.json
mv health-service-evidence.bound.json health-service-evidence.json
node connect-harness/tests/connectivity/health-service-evidence.mjs health-service-evidence.json --bound
- name: Upload only verified sanitized evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: connect-health-service-evidence-${{ github.run_id }}
path: health-service-evidence.json
retention-days: 14
if-no-files-found: error
@@ -49,23 +49,23 @@ permissions:
jobs:
performance-drive:
name: Verify native Linux x86_64 performance.drive service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,7 +76,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +122,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -183,7 +183,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('performance-drive-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-performance-drive-evidence-${{ github.run_id }}
path: |
@@ -49,23 +49,23 @@ permissions:
jobs:
performance-network:
name: Verify native Linux x86_64 performance.network service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,7 +76,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +122,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -183,7 +183,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('performance-network-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-performance-network-evidence-${{ github.run_id }}
path: |
@@ -37,6 +37,10 @@ on:
description: Expected rustfs binary SHA-256
required: true
type: string
catalog_sha256:
description: Independently reviewed release CPU catalog SHA-256
required: true
type: string
connect_sha:
description: Exact 40-character RustFS Connect harness commit
required: true
@@ -49,23 +53,23 @@ permissions:
jobs:
profile-cpu:
name: Verify native Linux x86_64 profile.cpu service job
runs-on: ubuntu-latest
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,7 +80,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -101,20 +105,27 @@ jobs:
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
github_api() {
curl --fail --silent --show-error \
--header "Authorization: Bearer $GH_TOKEN" \
--header 'Accept: application/vnd.github+json' \
"https://api.github.com/$1"
}
run=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
[[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]]
[[ $(jq -r '.head_branch' <<<"$run") == main ]]
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
[.jobs[] | select(.name == $name)] as $matches
| (($matches | length) == 1 and $matches[0].conclusion == "success")
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
artifact=$(github_api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
[[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]]
[[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]]
[[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]]
@@ -122,31 +133,39 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
run-id: ${{ inputs.build_run_id }}
github-token: ${{ github.token }}
- name: Extract exact RustFS binary
- name: Extract exact RustFS binary and reviewed catalog
shell: bash
env:
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CATALOG_SHA256: ${{ inputs.catalog_sha256 }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ "$CATALOG_SHA256" =~ ^[0-9a-f]{64}$ ]]
short_sha=${SOURCE_SHA:0:7}
package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit)
[[ -n "$package" ]]
mkdir -p binary
unzip -qq "$package" rustfs -d binary
unzip -qq "$package" rustfs rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256 -d binary
chmod +x binary/rustfs
printf '%s %s\n' "$BINARY_SHA256" binary/rustfs | sha256sum --check --strict
# The packaged digest is corroboration, never the independent trust input.
[[ $(cat binary/rustfs.cpu-symbol-catalog.sha256) == "$CATALOG_SHA256" ]]
printf '%s %s\n' "$CATALOG_SHA256" binary/rustfs.cpu-symbol-catalog.json | sha256sum --check --strict
chmod 0444 binary/rustfs.cpu-symbol-catalog.json
- name: Run signed service job acceptance
shell: bash
env:
CATALOG_SHA256: ${{ inputs.catalog_sha256 }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
@@ -156,6 +175,8 @@ jobs:
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \
CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \
CONNECT_E2E_PROFILE_EVIDENCE="$GITHUB_WORKSPACE/profile-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=profile
@@ -183,7 +204,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('profile-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-profile-cpu-evidence-${{ github.run_id }}
path: |
@@ -0,0 +1,235 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Connect service memory profile acceptance
on:
workflow_dispatch:
inputs:
build_run_id:
description: Successful main-branch Build and Release run ID
required: true
type: string
artifact_id:
description: Linux x86_64 GNU artifact from that run
required: true
type: string
source_sha:
description: Exact RustFS source commit
required: true
type: string
artifact_digest:
description: GitHub artifact digest including sha256 prefix
required: true
type: string
binary_sha256:
description: Independently verified rustfs binary SHA-256
required: true
type: string
connect_sha:
description: Exact Connect memory acceptance harness commit
required: true
type: string
permissions:
actions: read
contents: read
jobs:
profile-memory:
name: Verify signed service memory profile over mTLS
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/connect
path: connect-harness
persist-credentials: false
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Verify official source and artifact identity
shell: bash
env:
GH_TOKEN: ${{ github.token }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
SOURCE_SHA: ${{ inputs.source_sha }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
run: |
set -euo pipefail
[[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
[[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
jq -e --arg source "$SOURCE_SHA" '
.head_sha == $source and .head_branch == "main"
and .head_repository.full_name == "rustfs/rustfs"
and .name == "Build and Release" and .path == ".github/workflows/build.yml"
and .status == "completed" and .conclusion == "success"
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
.workflow_run.id == $run and .workflow_run.head_sha == $source
and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
and .digest == $digest and .expired == false
and (.expires_at | fromdateiso8601) > now
and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
' <<<"$artifact" >/dev/null
jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
[[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
printf '%s official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
- name: Extract only the verified official service binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
run: |
set -euo pipefail
python3 - <<'PY'
import io
import os
import pathlib
import shutil
import stat
import zipfile
def regular(entry):
mode = entry.external_attr >> 16
return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
with zipfile.ZipFile('official-artifact.zip') as outer:
entries = outer.infolist()
assert 1 <= len(entries) <= 16
assert len({entry.filename for entry in entries}) == len(entries)
assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
assert sum(entry.file_size for entry in entries) <= 2147483648
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
shutil.copyfileobj(source, destination)
PY
chmod 0755 binary/rustfs
printf '%s binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
file binary/rustfs | grep -Eq 'ELF 64-bit LSB.*x86-64'
- name: Run the real service job under S3 workload
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
git -C connect-harness diff --exit-code
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
changed=$(git -C connect-harness diff --name-only)
[[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/profile-memory-evidence.test.sh)
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PROFILE_MEMORY_EVIDENCE="$GITHUB_WORKSPACE/profile-memory-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=profile-memory
- name: Bind and validate sanitized evidence
shell: bash
env:
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
jq -e --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" '
.provenance.connectSha == $connect and .provenance.sourceSha == $source
and .provenance.binarySha256 == $binary
' profile-memory-evidence.json >/dev/null
jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
--arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
--arg expiry "$(cat artifact-expires-at)" '
.provenance += {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry}
' profile-memory-evidence.json > profile-memory-evidence.bound.json
mv profile-memory-evidence.bound.json profile-memory-evidence.json
node connect-harness/tests/connectivity/profile-memory-evidence.mjs profile-memory-evidence.json --bound
- name: Upload only verified sanitized evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: connect-profile-memory-evidence-${{ github.run_id }}
path: profile-memory-evidence.json
retention-days: 14
if-no-files-found: error
@@ -17,6 +17,14 @@ name: Connect profile.threads service-job acceptance
on:
workflow_dispatch:
inputs:
thread_scope:
description: Thread profile scope
required: true
type: choice
default: native
options:
- native
- runtime
build_run_id:
description: Main-branch Build and Release workflow run ID
required: true
@@ -48,24 +56,24 @@ permissions:
jobs:
profile-threads:
name: Verify native Linux x86_64 profile.threads service job
runs-on: ubuntu-latest
name: Verify Linux x86_64 profile.threads service job
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,10 +84,43 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Verify source run and artifact identity
shell: bash
env:
@@ -107,7 +148,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +163,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -148,8 +189,14 @@ jobs:
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
THREAD_SCOPE: ${{ inputs.thread_scope }}
run: |
set -euo pipefail
case "$THREAD_SCOPE" in
native) export CONNECT_E2E_THREAD_SCOPE=NATIVE_THREADS ;;
runtime) export CONNECT_E2E_THREAD_SCOPE=TOKIO_RUNTIME ;;
*) exit 1 ;;
esac
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
@@ -168,6 +215,7 @@ jobs:
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
THREAD_SCOPE: ${{ inputs.thread_scope }}
run: |
set -euo pipefail
jq \
@@ -177,13 +225,14 @@ jobs:
--arg artifactDigest "$ARTIFACT_DIGEST" \
--arg connectSha "$CONNECT_SHA" \
--arg sourceSha "$SOURCE_SHA" \
'. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha}' \
--arg requestedThreadScope "$THREAD_SCOPE" \
'. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha, requestedThreadScope: $requestedThreadScope}' \
profile-threads-service-job-evidence.json >profile-threads-service-job-evidence.bound.json
mv profile-threads-service-job-evidence.bound.json profile-threads-service-job-evidence.json
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('profile-threads-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-profile-threads-evidence-${{ github.run_id }}
path: |
@@ -49,23 +49,23 @@ permissions:
jobs:
top-api:
name: Verify native Linux x86_64 top.api service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,7 +76,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +122,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -183,7 +183,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('top-api-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-top-api-evidence-${{ github.run_id }}
path: |
+177 -68
View File
@@ -12,21 +12,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.
name: Connect top.disk artifact acceptance
name: Connect top.disk service-job acceptance
on:
workflow_dispatch:
inputs:
build_run_id:
description: Successful main-branch Build and Release workflow run ID
description: Successful main-branch Build and Release run ID
required: true
type: string
artifact_id:
description: Linux x86_64 GNU artifact ID from that run
description: Linux x86_64 GNU artifact from that run
required: true
type: string
source_sha:
description: Exact 40-character source commit
description: Exact RustFS source commit
required: true
type: string
artifact_digest:
@@ -34,7 +34,11 @@ on:
required: true
type: string
binary_sha256:
description: Expected rustfs binary SHA-256
description: Independently verified rustfs binary SHA-256
required: true
type: string
connect_sha:
description: Exact Connect top.disk service-job acceptance harness commit
required: true
type: string
@@ -44,16 +48,67 @@ permissions:
jobs:
top-disk:
name: Verify native Linux x86_64 top.disk artifact
runs-on: ubuntu-latest
timeout-minutes: 15
name: Verify native top.disk service job over mTLS
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Checkout exact RustFS source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Verify source run and artifact identity
- name: Checkout exact Connect harness
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
persist-credentials: false
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Verify official source and artifact identity
shell: bash
env:
GH_TOKEN: ${{ github.token }}
@@ -61,82 +116,136 @@ jobs:
ARTIFACT_ID: ${{ inputs.artifact_id }}
SOURCE_SHA: ${{ inputs.source_sha }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
run: |
set -euo pipefail
[[ "$BUILD_RUN_ID" =~ ^[0-9]+$ ]]
[[ "$ARTIFACT_ID" =~ ^[0-9]+$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
[[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
jq -e --arg source "$SOURCE_SHA" '
.head_sha == $source and .head_branch == "main"
and .head_repository.full_name == "rustfs/rustfs"
and .name == "Build and Release" and .path == ".github/workflows/build.yml"
and .status == "completed" and .conclusion == "success"
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
.workflow_run.id == $run and .workflow_run.head_sha == $source
and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
and .digest == $digest and .expired == false
and (.expires_at | fromdateiso8601) > now
and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
' <<<"$artifact" >/dev/null
jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
[[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
printf '%s official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
[[ $(jq -r '.conclusion' <<<"$run") == success ]]
[[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]]
[[ $(jq -r '.head_branch' <<<"$run") == main ]]
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
[[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]]
[[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]]
[[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]]
[[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]]
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
run-id: ${{ inputs.build_run_id }}
github-token: ${{ github.token }}
- name: Extract exact RustFS binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
short_sha=${SOURCE_SHA:0:7}
package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit)
[[ -n "$package" ]]
mkdir -p binary
unzip -qq "$package" rustfs -d binary
chmod +x binary/rustfs
- name: Run top.disk acceptance
- name: Extract only the verified official service binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
run: |
set -euo pipefail
scripts/ci/check_connect_top_disk_artifact.sh \
binary/rustfs "$SOURCE_SHA" "$BINARY_SHA256" \
top-disk-runtime-evidence.json top-disk.zip
python3 - <<'PY'
import io
import os
import pathlib
import shutil
import stat
import zipfile
- name: Bind workflow and artifact provenance
def regular(entry):
mode = entry.external_attr >> 16
return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
with zipfile.ZipFile('official-artifact.zip') as outer:
entries = outer.infolist()
assert 1 <= len(entries) <= 16
assert len({entry.filename for entry in entries}) == len(entries)
assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
assert sum(entry.file_size for entry in entries) <= 2147483648
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
header = source.read(20)
assert header[:7] == b'\x7fELF\x02\x01\x01' and header[18:20] == b'\x3e\x00', 'Expected a little-endian ELF64 x86-64 binary'
destination.write(header)
shutil.copyfileobj(source, destination)
PY
chmod 0755 binary/rustfs
printf '%s binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
- name: Run the real top.disk service job
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
CONNECT_E2E_PRODUCER_SOURCE: ${{ inputs.source_sha }}
CONNECT_E2E_PRODUCER_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_E2E_PRODUCER_ARTIFACT_ID: ${{ inputs.artifact_id }}
run: |
set -euo pipefail
git -C connect-harness diff --exit-code
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
changed=$(git -C connect-harness diff --name-only)
[[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_DISK_EVIDENCE="$GITHUB_WORKSPACE/top-disk-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-disk
- name: Bind and validate sanitized evidence
shell: bash
env:
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
jq \
--arg workflowRunId "$GITHUB_RUN_ID" \
--arg buildRunId "$BUILD_RUN_ID" \
--arg artifactId "$ARTIFACT_ID" \
--arg artifactDigest "$ARTIFACT_DIGEST" \
'. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest}' \
top-disk-runtime-evidence.json >top-disk-runtime-evidence.bound.json
mv top-disk-runtime-evidence.bound.json top-disk-runtime-evidence.json
jq -e --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" --arg artifact "$ARTIFACT_ID" '
.status == "PASS" and .officialArtifactId == $artifact and .producerSource == $source and .producerSha256 == $binary
and .result.provenance.sourceCommit == $source
and .result.provenance.executableSha256 == $binary
' top-disk-service-job-evidence.json >/dev/null
jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
--arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
--arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" \
--arg expiry "$(cat artifact-expires-at)" '
. + {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry,connectSha:$connect,sourceSha:$source,binarySha256:$binary}
' top-disk-service-job-evidence.json > top-disk-service-job-evidence.bound.json
mv top-disk-service-job-evidence.bound.json top-disk-service-job-evidence.json
- name: Upload acceptance evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
- name: Upload only verified sanitized evidence
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-top-disk-evidence-${{ github.run_id }}
path: |
top-disk-runtime-evidence.json
top-disk.zip
name: connect-top-disk-service-job-evidence-${{ github.run_id }}
path: top-disk-service-job-evidence.json
retention-days: 14
if-no-files-found: error
@@ -49,23 +49,23 @@ permissions:
jobs:
top-locks:
name: Verify native Linux x86_64 top.locks service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,7 +76,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +122,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -183,7 +183,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('top-locks-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-top-locks-evidence-${{ github.run_id }}
path: |
@@ -49,23 +49,23 @@ permissions:
jobs:
top-rpc:
name: Verify native Linux x86_64 top.rpc service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,7 +76,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +122,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -183,7 +183,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('top-rpc-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-top-rpc-evidence-${{ github.run_id }}
path: |
+4 -4
View File
@@ -76,7 +76,7 @@ jobs:
NEXTEST_PROFILE: ci
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -89,7 +89,7 @@ jobs:
install-build-packaging-tools: 'false'
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
with:
tool: cargo-llvm-cov
@@ -112,7 +112,7 @@ jobs:
- name: Upload coverage artifact
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: coverage-lcov-${{ github.run_number }}
path: |
@@ -134,7 +134,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+26 -13
View File
@@ -47,7 +47,7 @@ on:
default: true
type: boolean
version:
description: "Version to build (latest for stable release, or specific version like v1.0.0, v1.0.0-alpha1)"
description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
required: false
default: "latest"
type: string
@@ -82,9 +82,8 @@ jobs:
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main' &&
!contains(github.event.workflow_run.head_branch, '-preview'))
runs-on: ubuntu-latest
github.event.workflow_run.head_branch != 'main')
runs-on: sm-standard-2
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
@@ -97,7 +96,7 @@ jobs:
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
@@ -197,7 +196,14 @@ jobs:
version="${version#v}"
fi
if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Building Docker image for preview: $version (version tags only)"
elif [[ "$version" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
@@ -249,11 +255,14 @@ jobs:
echo "🚀 Building with latest stable release version"
;;
*-preview*)
if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
echo "❌ Invalid preview version: $input_version" >&2
exit 1
fi
build_type="preview"
is_prerelease=true
should_build=false
should_push=false
echo "⏭️ Preview tags do not publish Docker images"
version="${input_version#v}"
echo "🔍 Building with preview version: $version (version tags only)"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
@@ -343,7 +352,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
@@ -492,7 +501,7 @@ jobs:
name: Scan Docker Images
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
permissions:
contents: read
@@ -535,7 +544,7 @@ jobs:
category: container-image-${{ matrix.variant }}
- name: Upload container scan report
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: container-image-scan-${{ matrix.variant }}
path: trivy-${{ matrix.variant }}.sarif
@@ -547,7 +556,7 @@ jobs:
name: Docker Build Summary
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Docker build completion summary
@@ -563,6 +572,10 @@ jobs:
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview Docker image has been built with ${VERSION} tags"
echo "⏭️ Preview images do not update latest or prerelease channels"
;;
"release")
echo "🚀 Release Docker image has been built with ${VERSION} tags"
echo "✅ This image is ready for production use"
+4 -3
View File
@@ -38,6 +38,7 @@ on:
- "Cargo.lock"
- "Cargo.toml"
- ".config/nextest.toml"
- ".config/e2e-distributed-selection.txt"
- ".github/workflows/e2e-distributed.yml"
- "crates/audit/**"
- "crates/common/**"
@@ -93,7 +94,7 @@ jobs:
UPGRADE_SOURCE_SHA256: 7c789386bf85278f865b8e0d359bf4edb84d5aa408cc3fa54a18c25ca74cd6e7
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -174,7 +175,7 @@ jobs:
- name: Upload distributed e2e diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-distributed-${{ github.run_number }}
path: |
@@ -206,7 +207,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
@@ -63,7 +63,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -77,7 +77,7 @@ jobs:
# The STS dual-node test requires awscurl and fails if it is unavailable.
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -108,7 +108,7 @@ jobs:
- name: Upload nextest junit report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-replication-nightly-junit-${{ github.run_number }}
path: |
@@ -126,7 +126,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -157,7 +157,7 @@ jobs:
- name: Upload cluster fault diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-cluster-nightly-${{ github.run_number }}
path: |
@@ -176,7 +176,7 @@ jobs:
RUSTFS_BUILD_FEATURES: ftps,webdav,sftp
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -213,7 +213,7 @@ jobs:
- name: Upload protocol diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-protocol-nightly-${{ github.run_number }}
path: |
@@ -236,7 +236,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+14 -9
View File
@@ -159,7 +159,7 @@ jobs:
TEST_MODE: ${{ matrix.test-mode }}
S3_SHARD_INDEX: ${{ matrix.shard-index }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -167,7 +167,7 @@ jobs:
# ship a working pip (ci-1: a bare python3 without pip is what broke the
# scheduled sweep). Keeps the workflow correct across runner drift.
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -264,7 +264,7 @@ jobs:
hostname: rustfs4
volumes: [rustfs4-data:/data]
lb:
image: haproxy:2.9
image: haproxy:3.4.4
hostname: lb
networks: [rustfs-net]
ports:
@@ -308,6 +308,11 @@ jobs:
docker compose -f compose.yml up -d
# Validate the pinned HAProxy image parses the exact config before
# the S3 compatibility sweep starts.
docker compose -f compose.yml exec -T lb \
haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg
- name: Wait for RustFS ready
run: |
for _ in {1..120}; do
@@ -357,7 +362,7 @@ jobs:
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: s3tests-${{ env.TEST_MODE }}-shard-${{ matrix.shard-index }}
path: artifacts/**
@@ -366,15 +371,15 @@ jobs:
name: Upstream HEAD classification canary
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
continue-on-error: true
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 20
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -435,7 +440,7 @@ jobs:
- name: Upload canary artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: s3tests-upstream-head
path: artifacts/s3tests-upstream-head/**
@@ -454,7 +459,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+66 -21
View File
@@ -51,8 +51,53 @@ env:
UPGRADE_SOURCE_SHA256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d
jobs:
build:
name: Build upgrade candidate
runs-on: sm-standard-2
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-shared-key: e2e-upgrade-server
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Download pinned previous release
run: |
mkdir -p target/debug
archive="target/debug/$UPGRADE_SOURCE_ASSET"
curl --fail --location --retry 3 --output "$archive" \
"https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
echo "$UPGRADE_SOURCE_SHA256 $archive" | sha256sum --check --strict
- name: Build current RustFS binary
run: python3 scripts/e2e_binary.py build
- name: Upload verified upgrade candidate
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: upgrade-candidate-${{ github.run_id }}
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
target/debug/${{ env.UPGRADE_SOURCE_ASSET }}
if-no-files-found: error
retention-days: 3
compression-level: 0
overwrite: true
upgrade:
name: ${{ matrix.name }}
needs: build
strategy:
fail-fast: false
matrix:
@@ -63,59 +108,61 @@ jobs:
# the CI required-check names. UPGRADE_SOURCE_VERSION above is the
# single source of truth for which release they actually run against.
- name: Direct upgrade from the previous release
cache_key: e2e-direct-upgrade
test: direct_upgrade_from_rc2_preserves_object_contracts
artifact: direct-upgrade
- name: Mixed-version rolling upgrade from the previous release
cache_key: e2e-mixed-version-upgrade
test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
artifact: mixed-version-upgrade
- name: Bucket configuration survives the upgrade
cache_key: e2e-bucket-config-upgrade
test: direct_upgrade_from_previous_release_preserves_bucket_configuration
artifact: bucket-config-upgrade
- name: Rollback reads current bucket metadata
cache_key: e2e-bucket-config-rollback
test: rollback_to_previous_release_reads_current_bucket_metadata
artifact: bucket-config-rollback
- name: ODM configuration recovery after rc.5 rollback
cache_key: e2e-odm-config-rollback
test: rc5_rollback_requires_restoring_odm_configuration
artifact: odm-config-rollback
- name: Multipart layouts survive the rc.5 upgrade
cache_key: e2e-multipart-layout-upgrade
test: direct_upgrade_from_rc5_preserves_multipart_layouts
artifact: multipart-layout-upgrade
- name: rc.5 multipart replication baseline
cache_key: e2e-multipart-layout-baseline
test: rc5_baseline_replicates_multipart_layouts
artifact: multipart-layout-baseline
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-shared-key: ${{ matrix.cache_key }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
cache-shared-key: e2e-upgrade-tests
# One main-branch consumer warms the shared test-harness dependencies.
cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Download pinned previous release
- name: Download verified upgrade candidate
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: upgrade-candidate-${{ github.run_id }}
path: target/debug
- name: Restore candidate executable bit
run: chmod +x target/debug/rustfs
- name: Verify and unpack pinned previous release
env:
SOURCE_DIR: ${{ runner.temp }}/rustfs-upgrade-source
run: |
set -euo pipefail
mkdir -p "$SOURCE_DIR"
archive="$SOURCE_DIR/$UPGRADE_SOURCE_ASSET"
curl --fail --location --retry 3 --output "$archive" \
"https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
archive="target/debug/$UPGRADE_SOURCE_ASSET"
echo "$UPGRADE_SOURCE_SHA256 $archive" | sha256sum --check --strict
unzip -q "$archive" -d "$SOURCE_DIR"
chmod +x "$SOURCE_DIR/rustfs"
@@ -123,21 +170,19 @@ jobs:
echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
- name: Build current RustFS binary
run: |
python3 scripts/e2e_binary.py build
- name: Run upgrade compatibility test
env:
RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
NO_PROXY: 127.0.0.1,localhost,::1
run: |
export no_proxy="$NO_PROXY"
python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
"upgrade_compatibility_test::${{ matrix.test }}" \
-- --ignored --exact --nocapture
- name: Upload scanner/heal G09 evidence
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
@@ -146,7 +191,7 @@ jobs:
- name: Upload server logs
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-logs
+35 -3
View File
@@ -11,12 +11,44 @@ concurrency:
cancel-in-progress: false
jobs:
collect:
runs-on: ubuntu-latest
runs-on: smoke-testing
timeout-minutes: 15
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Collect and publish verified chain health
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
@@ -24,7 +56,7 @@ jobs:
run: python3 scripts/functional_chain_health.py --source-ref "${NIGHTLY_SOURCE_REF}" --publish --output "${RUNNER_TEMP}/chain-health.json"
- name: Retain health observation
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-health-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-health.json
+11 -11
View File
@@ -60,7 +60,7 @@ jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Explain cancellation run
@@ -81,7 +81,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -93,7 +93,7 @@ jobs:
cache-save-if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }}
- name: Install cargo-fuzz
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
with:
tool: cargo-fuzz
@@ -113,7 +113,7 @@ jobs:
done
- name: Upload prebuilt fuzz binaries
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: fuzz-prebuilt-binaries-${{ github.run_number }}
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/
@@ -143,12 +143,12 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: fuzz-prebuilt-binaries-${{ github.run_number }}
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
@@ -167,7 +167,7 @@ jobs:
- name: Upload fuzz smoke artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: fuzz-smoke-${{ matrix.target }}-${{ github.run_number }}
path: |
@@ -197,12 +197,12 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: fuzz-prebuilt-binaries-${{ github.run_number }}
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
@@ -221,7 +221,7 @@ jobs:
- name: Upload nightly fuzz artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: fuzz-nightly-${{ matrix.target }}-${{ github.run_number }}
path: |
@@ -246,7 +246,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+45 -8
View File
@@ -31,7 +31,7 @@ permissions:
jobs:
build-helm-package:
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
@@ -48,9 +48,10 @@ jobs:
chart_version: ${{ steps.version.outputs.chart_version }}
steps:
- name: Checkout helm chart repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Checkout release source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
persist-credentials: false
# Both inputs reach the shell through env rather than `${{ }}`
@@ -83,6 +84,40 @@ jobs:
./scripts/helm_chart_version.sh "$RAW_TAG"
# Docker and Helm share the binary-build trigger. Do not publish a chart
# until the image selected by its default appVersion is pullable.
- name: Wait for release image
timeout-minutes: 20
env:
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -euo pipefail
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
[[ "$APP_VERSION" == *-preview* ]]; then
echo "Invalid deliverable image version: $APP_VERSION" >&2
exit 1
fi
for attempt in {1..60}; do
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
| jq -er '.token // empty') &&
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
--header "Authorization: Bearer $token" \
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
exit 0
fi
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
if (( attempt < 60 )); then
sleep 15
fi
done
echo "Release image is unavailable; refusing to publish Helm chart" >&2
exit 1
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
@@ -107,21 +142,23 @@ jobs:
--version "${{ steps.version.outputs.chart_version }}"
- name: Upload helm package as artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: helm-package
path: helm/rustfs/*.tgz
path: |
helm/rustfs/*.tgz
helm/rustfs/README.md
retention-days: 1
publish-helm-package:
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
steps:
- name: Checkout helm package repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
@@ -129,7 +166,7 @@ jobs:
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: helm-package
path: ./
+1 -1
View File
@@ -32,7 +32,7 @@ permissions:
jobs:
build:
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- uses: usthe/issues-translate-action@b41f55ddc81d7d54bd542a4f289fe28ec081898e # v2.7
+3 -3
View File
@@ -60,7 +60,7 @@ permissions:
jobs:
minio-interop:
name: MinIO interop (EC + SSE read parity)
# Skip on forks: needs the repo's runners and is not a contributor gate.
# Keep this scheduled interoperability lane scoped to the upstream repo.
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 40
@@ -83,7 +83,7 @@ jobs:
INTEROP_REQUIRED_TESTS: '["reads_minio_generated_sse_s3_multipart_fixture", "reads_minio_generated_sse_kms_multipart_fixture", "rejects_minio_generated_sse_s3_fixture_with_wrong_kms_key", "rejects_minio_generated_sse_s3_fixture_with_truncated_ciphertext"]'
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -137,7 +137,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+4 -4
View File
@@ -114,10 +114,10 @@ jobs:
# label was abandoned (no docker.sock on the pod its only run landed on).
# TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
# run) is added by the ci-8 composite action; do not implement it here.
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 120
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -246,7 +246,7 @@ jobs:
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: mint
path: artifacts/**
@@ -264,7 +264,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+41 -74
View File
@@ -44,14 +44,14 @@ env:
jobs:
resolve-source:
name: Resolve nightly source
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
outputs:
source_sha: ${{ steps.source.outputs.sha }}
source_ref: ${{ steps.source.outputs.ref }}
steps:
- name: Checkout selected source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ env.NIGHTLY_BUILD_REF }}
@@ -71,7 +71,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.resolve-source.outputs.source_sha }}
@@ -117,10 +117,15 @@ jobs:
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://rustfs.com/docs/ for more information
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ROOT_USER=""
# RUSTFS_ROOT_PASSWORD=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
# dpkg versions must start with a digit and cannot contain hyphens;
@@ -146,37 +151,13 @@ jobs:
/etc/default/rustfs
CONFFILES
cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
#!/bin/bash
set -e
if ! getent passwd rustfs > /dev/null 2>&1; then
useradd -r -s /bin/false -d /opt/rustfs rustfs
fi
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
POSTINST
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
systemctl stop rustfs
fi
PRERM
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTRM
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
@@ -214,10 +195,15 @@ jobs:
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://rustfs.com/docs/ for more information
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ROOT_USER=""
# RUSTFS_ROOT_PASSWORD=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
@@ -231,35 +217,13 @@ jobs:
--description "High-performance distributed object storage" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--after-install <(cat << 'POSTINST'
#!/bin/bash
set -e
if ! getent passwd rustfs > /dev/null 2>&1; then
useradd -r -s /bin/false -d /opt/rustfs rustfs
fi
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTINST
) \
--before-remove <(cat << 'PRERM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
systemctl stop rustfs
fi
PRERM
) \
--after-remove <(cat << 'POSTRM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTRM
) \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
"rustfs-nightly-${DEB_DATE}/usr/bin/rustfs=/usr/bin/rustfs" \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
@@ -273,12 +237,12 @@ jobs:
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
- name: Upload DEB artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ steps.deb.outputs.deb_file }}
path: ${{ steps.deb.outputs.deb_file }}
- name: Upload RPM artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ steps.rpm.outputs.rpm_file }}
path: ${{ steps.rpm.outputs.rpm_file }}
@@ -364,7 +328,7 @@ jobs:
- name: Upload nightly candidate manifest
if: ${{ steps.publish.outputs.candidate_file != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: nightly-candidate-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ steps.publish.outputs.candidate_file }}
@@ -396,9 +360,12 @@ jobs:
# live scripts (AppRole login, three-node Raft leader failover).
#
# GitHub-hosted ubuntu-latest, deliberately not the self-hosted sm-standard
# fleet: the HA failover script needs a working Docker daemon, and the
# self-hosted fleet is heterogeneous — a docker-dependent workflow has been
# burned by it before (see the banner in e2e-s3tests.yml, rustfs/backlog#1149).
# fleet: both lanes need a working Docker daemon, and the sm-standard-2 pool
# does not provide one. #8112 rerouted these lanes there and both died in
# seconds on "Docker is not available" while the package itself built and
# published fine, which kept the nightly chain from firing (its gate
# requires a successful build run). See also the banner in e2e-s3tests.yml
# and rustfs/backlog#1149.
kms-vault-lane:
needs: resolve-source
name: KMS live Vault lane
@@ -419,7 +386,7 @@ jobs:
NO_PROXY: 127.0.0.1,localhost
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.resolve-source.outputs.source_sha }}
@@ -509,7 +476,7 @@ jobs:
NO_PROXY: 127.0.0.1,localhost
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.resolve-source.outputs.source_sha }}
@@ -537,7 +504,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+6 -6
View File
@@ -33,30 +33,30 @@ concurrency:
jobs:
update-flake:
name: Update flake.lock
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
# persist-credentials-exempt: update-flake-lock pushes the branch and opens
# the PR. It passes FLAKE_UPDATE_TOKEN to create-pull-request itself rather
# than reusing .git/config, but that is unverified — exempt until a
# workflow_dispatch run confirms it (rustfs/backlog#1602).
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@629b284231c2a82554b724e357e47fc6020833c8 # v3
uses: DeterminateSystems/determinate-nix-action@8d87e8d5e5b8a8309d4281094560f127d9a265f1 # v3.22.5
- name: Cache Nix
uses: DeterminateSystems/flakehub-cache-action@1f9a51a2959d3e26c7838c6f3bf9f48acae525ea # v3.20.0
uses: DeterminateSystems/flakehub-cache-action@83282a8aef353db1d659d3bacf4a28b6683de0b3 # v3.22.5
- name: Check Nix flake inputs
uses: DeterminateSystems/flake-checker-action@3164002371bc90729c68af0e24d5aacf20d7c9f6 # v12
uses: DeterminateSystems/flake-checker-action@786422608c7bded2bbc9741ad9f91356842bf520 # v14
- name: Update flake.lock
id: update
uses: DeterminateSystems/update-flake-lock@fd9359ac79d0e912f1b4b947a48470b3e2799b56 # main
uses: DeterminateSystems/update-flake-lock@da03c0f078bc4b2c37ee4f7e072d34bf8f188bb3 # v29
with:
git-author-name: houseme
git-author-email: housemecn@gmail.com
+6 -6
View File
@@ -54,7 +54,7 @@ jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Explain cancellation run
@@ -63,7 +63,7 @@ jobs:
nix-validation:
name: Nix Build & Check
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 60
permissions:
contents: read
@@ -72,12 +72,12 @@ jobs:
NIX_CURL_FLAGS: -A cargo/stable
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@4eea0b33e3d1f02ecfe37cf16e7204c424009606 # v3.21.0
uses: DeterminateSystems/determinate-nix-action@8d87e8d5e5b8a8309d4281094560f127d9a265f1 # v3.22.5
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
extra-conf: |
@@ -86,10 +86,10 @@ jobs:
max-jobs = 1
- name: Cache Nix
uses: DeterminateSystems/flakehub-cache-action@c01e819d047464c3edf6ba778f075952af5a3aa7 # v3.21.0
uses: DeterminateSystems/flakehub-cache-action@83282a8aef353db1d659d3bacf4a28b6683de0b3 # v3.22.5
- name: Check Nix Flake Inputs
uses: DeterminateSystems/flake-checker-action@3164002371bc90729c68af0e24d5aacf20d7c9f6 # v12
uses: DeterminateSystems/flake-checker-action@786422608c7bded2bbc9741ad9f91356842bf520 # v14
with:
fail-mode: true
ignore-missing-flake-lock: false
+8 -3
View File
@@ -57,7 +57,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -72,6 +72,11 @@ jobs:
- name: Build RustFS
run: cargo build --locked -p rustfs --bin rustfs
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install pinned request signer
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
@@ -82,7 +87,7 @@ jobs:
- name: Upload service logs
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: oidc-keycloak-live-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-keycloak-live-*/**/*.log
@@ -101,7 +106,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
@@ -69,8 +69,7 @@ env:
jobs:
minio-source:
name: MinIO source (read-through, list-through, backfill)
# Skip on forks: needs this repository's runners and is not a contributor
# gate.
# Keep this scheduled interoperability lane scoped to the upstream repo.
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 90
@@ -93,7 +92,7 @@ jobs:
NEXTEST_LISTING: ${{ github.workspace }}/artifacts/odm-interop/minio/selection.json
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -108,11 +107,15 @@ jobs:
run: |
set -euo pipefail
mkdir -p artifacts/odm-interop/minio
docker build -t rustfs-odm-interop-minio \
-f crates/rio-v2/tests/minio_fixture_lab/Dockerfile \
crates/rio-v2/tests/minio_fixture_lab
docker run -d --name rustfs-odm-interop-minio \
--entrypoint /usr/local/bin/minio \
-e "MINIO_ROOT_USER=${MINIO_ROOT_USER}" \
-e "MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}" \
-p 9100:9000 \
minio/minio:RELEASE.2025-09-07T16-13-09Z server /data
rustfs-odm-interop-minio server /data
for _ in $(seq 1 120); do
curl -fsS http://127.0.0.1:9100/minio/health/live >/dev/null 2>&1 && break
sleep 1
@@ -166,7 +169,7 @@ jobs:
- name: Upload the MinIO interop report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: odm-interop-minio-${{ github.run_number }}-${{ github.run_attempt }}
path: |
@@ -183,7 +186,7 @@ jobs:
cloud-source:
name: ${{ matrix.provider }} source (three-case minimum)
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
strategy:
fail-fast: false
@@ -237,7 +240,7 @@ jobs:
- name: Checkout repository
if: steps.credentials.outputs.present == 'true'
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -286,7 +289,7 @@ jobs:
- name: Upload the ${{ matrix.provider }} interop report
if: always() && steps.credentials.outputs.present == 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: odm-interop-${{ matrix.provider }}-${{ github.run_number }}-${{ github.run_attempt }}
path: |
@@ -306,7 +309,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+95 -88
View File
@@ -25,9 +25,10 @@
#
# Flow:
# 1. Resolve and validate the selected Build workflow run and source identity
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu) from build artifacts
# 3. Build DEB packages for amd64 and arm64
# 4. Build RPM packages for x86_64 and aarch64
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
# aarch64-musl) from build artifacts
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
@@ -76,7 +77,7 @@ jobs:
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
@@ -122,7 +123,7 @@ jobs:
semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
if [[ -n "$TAG" ]]; then
if [[ "$TAG" =~ ^${semver_core}-${prerelease_id}-preview\.(0|[1-9][0-9]*)$ ]]; then
if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
BUILD_TYPE=preview
elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
BUILD_TYPE=prerelease
@@ -224,11 +225,11 @@ jobs:
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Build DEB and RPM packages for each architecture
# Build DEB and RPM packages for each architecture and libc variant
package:
name: Package (${{ matrix.arch }})
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: resolve
runs-on: ubuntu-latest
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
strategy:
fail-fast: false
@@ -236,16 +237,38 @@ jobs:
matrix:
include:
- arch: x86_64
libc: gnu
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
# gnu binaries dynamically link glibc; musl binaries are static
# and run without a libc dependency.
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: aarch64
libc: gnu
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: x86_64
libc: musl
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-musl"
deb_depends: ""
rpm_depends: ""
- arch: aarch64
libc: musl
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-musl"
deb_depends: ""
rpm_depends: ""
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -256,17 +279,17 @@ jobs:
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
DEB_ARCH: ${{ matrix.deb_arch }}
RPM_ARCH: ${{ matrix.rpm_arch }}
ARCH: ${{ matrix.rpm_arch }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
normalized=$(./scripts/release/package_versions.sh \
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$DEB_ARCH" "$RPM_ARCH")
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
- name: Download binary artifact from build run
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
@@ -308,6 +331,8 @@ jobs:
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
DEB_ARCH: ${{ matrix.deb_arch }}
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
DEB_DEPENDS: ${{ matrix.deb_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
@@ -328,10 +353,15 @@ jobs:
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://rustfs.com/docs/ for more information
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ROOT_USER=""
# RUSTFS_ROOT_PASSWORD=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
@@ -340,9 +370,17 @@ jobs:
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
Depends: libc6 (>= 2.31)
EOF
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
if [[ -n "$DEB_DEPENDS" ]]; then
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
fi
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage
Description: High-performance distributed object storage (${LIBC} build)
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
@@ -354,37 +392,13 @@ jobs:
/etc/default/rustfs
CONFFILES
cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
#!/bin/bash
set -e
if ! getent passwd rustfs > /dev/null 2>&1; then
useradd -r -s /bin/false -d /opt/rustfs rustfs
fi
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
POSTINST
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
systemctl stop rustfs
fi
PRERM
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTRM
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
@@ -408,24 +422,38 @@ jobs:
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
RPM_ARCH: ${{ matrix.rpm_arch }}
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
RPM_DEPENDS: ${{ matrix.rpm_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
echo "Building RPM for ${RPM_ARCH}"
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
sudo gem install fpm
./scripts/test_package_versions.sh --require-package-managers
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
depends_args=()
if [[ -n "$RPM_DEPENDS" ]]; then
depends_args+=(--depends "$RPM_DEPENDS")
fi
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://rustfs.com/docs/ for more information
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ROOT_USER=""
# RUSTFS_ROOT_PASSWORD=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
@@ -434,40 +462,18 @@ jobs:
--iteration "$RPM_RELEASE" \
--architecture "$RPM_ARCH" \
--package "$RPM_FILE" \
--depends "glibc >= 2.31" \
"${depends_args[@]}" \
--description "High-performance distributed object storage (${LIBC} build)" \
--maintainer "RustFS Team <support@rustfs.com>" \
--description "High-performance distributed object storage" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--after-install <(cat <<'POSTINST'
#!/bin/bash
set -e
if ! getent passwd rustfs > /dev/null 2>&1; then
useradd -r -s /bin/false -d /opt/rustfs rustfs
fi
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTINST
) \
--before-remove <(cat <<'PRERM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
systemctl stop rustfs
fi
PRERM
) \
--after-remove <(cat <<'POSTRM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTRM
) \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
@@ -489,16 +495,16 @@ jobs:
echo "✅ RPM built: $RPM_FILE"
- name: Upload packages to artifacts
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.arch }}
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
path: |
*.deb
*.rpm
retention-days: 30
- name: Upload packages to Cloudflare R2
if: env.R2_ACCESS_KEY_ID != ''
if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
@@ -587,10 +593,11 @@ jobs:
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# GitHub stores release asset names with '~' normalized to '.'
# (e.g. rustfs_1.0.0~rc.2_amd64.deb is stored as
# rustfs_1.0.0.rc.2_amd64.deb), so checksum entries must
# reference the name as stored on the release.
# GitHub stores release asset names with '~' normalized to '.'.
# Current package file names embed the raw tag and contain no
# '~', but older releases may still ship such assets, so the
# checksum entries must reference the name as stored on the
# release.
github_base="${base//\~/.}"
# Remove any stale entry (both naming variants), then append
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
@@ -612,7 +619,7 @@ jobs:
name: Summary
needs: [ resolve, package ]
if: always()
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 5
steps:
- name: Print summary
+4 -4
View File
@@ -57,7 +57,7 @@ jobs:
timeout-minutes: 180
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0 # baseline may be an earlier successful scheduled head
@@ -94,7 +94,7 @@ jobs:
- name: Find last successful scheduled baseline
id: scheduled_baseline
if: github.event_name == 'schedule'
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
result-encoding: string
script: |
@@ -271,7 +271,7 @@ jobs:
- name: Upload A/B results
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: hotpath-warp-ab-${{ github.run_number }}
# Includes per-cell median_summary.csv / baseline_compare.csv, both gates,
@@ -372,7 +372,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+3 -3
View File
@@ -43,11 +43,11 @@ concurrency:
jobs:
check-ephemerality:
name: Check runner ephemerality
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -72,7 +72,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
@@ -0,0 +1,221 @@
# RustFS Fault-Tolerance (degradation) Matrix Sweep
#
# Exhaustive topology x EC outage sweep, complementary to the scenario suite
# (rustfs-fault-tolerance-test.yml). For each topology (nodes x drives per
# node) and each EC level P in 1..N*drives/2 the runner installs that exact
# cluster shape and drives progressive node outages plus recovery:
#
# M-<topo>EC<P>-full full-capacity reads/writes must succeed
# M-<topo>EC<P>-n<k> with k nodes online: reads succeed iff the online
# drives cover the read quorum (N-P), writes iff they
# cover the write quorum (N-P+1); probes below quorum
# assert the expected outage
# M-<topo>EC<P>-rec after restoring every node, reads/writes succeed
#
# 9 topologies x EC levels = 36 combinations, 300 cases. The full sweep runs
# 3-5 hours and is dispatched on demand (e.g. after a nightly chain goes
# green), not nightly. Case ids and expectations come from product source
# (default_parity_count, erasure set sizing) via rustfs/auto-testing.
name: RustFS Fault-Tolerance Matrix
on:
workflow_dispatch:
inputs:
package_url:
description: 'Direct .deb URL. Required unless the nightly default is wanted.'
required: false
type: string
auto_testing_ref:
description: 'auto-testing branch, tag, or SHA (defaults to main)'
required: false
default: main
type: string
matrix_topo:
description: 'Topology filter like 4x2; empty sweeps all nine topologies'
required: false
default: ''
type: string
matrix_ec:
description: 'EC/parity filter; empty sweeps every EC level'
required: false
default: ''
type: string
cleanup_before:
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
type: boolean
default: true
cleanup_after:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
permissions:
contents: read
# The suite stops services and revokes drive access on the shared fleet; only one
# functional suite may touch the environment at a time.
concurrency:
group: rustfs-shared-functional-tests-v2
cancel-in-progress: false
defaults:
run:
shell: bash
env:
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
jobs:
fault-tolerance-matrix:
runs-on: smoke-testing
# The full 38-combination sweep takes 3-5h; 360 is the runner job ceiling.
timeout-minutes: 360
steps:
- name: Initialize functional evidence
id: evidence
run: |
set -euo pipefail
umask 077
FUNCTIONAL_ARTIFACTS_DIR="${RUNNER_TEMP}/rustfs-fault-tolerance-matrix-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
mkdir -- "${FUNCTIONAL_ARTIFACTS_DIR}" "${FUNCTIONAL_ARTIFACTS_DIR}-scratch" "${FUNCTIONAL_ARTIFACTS_DIR}/evidence"
{
printf 'FUNCTIONAL_ARTIFACTS_DIR=%s\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'LOG_FILE=%s/suite.log\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'REPORT_FILE=%s/report.md\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'RESULTS_FILE=%s/results.json\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'EVIDENCE_DIR=%s/evidence\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
} >> "${GITHUB_ENV}"
- name: Checkout auto-testing scripts
timeout-minutes: 5
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ inputs.auto_testing_ref }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
path: auto-testing
persist-credentials: false
- name: Record auto-testing revision
run: |
set -euo pipefail
AUTO_TESTING_SHA="$(git -C auto-testing rev-parse HEAD)"
printf 'AUTO_TESTING_SHA=%s\n' "${AUTO_TESTING_SHA}" >> "${GITHUB_ENV}"
echo "auto-testing revision: ${AUTO_TESTING_SHA}"
- name: Show environment
run: |
uname -a
jq --version
aws --version
echo "auto-testing: ${AUTO_TESTING_SHA}"
df -h /data | tail -1
- name: Cleanup environment (before)
timeout-minutes: 5
if: ${{ inputs.cleanup_before != 'false' }}
run: |
./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}"
- name: Run matrix sweep (topology x EC outage combinations)
timeout-minutes: 350
id: test
# No continue-on-error: an UNEXPECTED verdict (a probe that should
# fail by quorum math but passes, or vice versa) is the finding this
# sweep exists to surface, and the red run is the signal until matrix
# runs get backlog integration like the scenario suite has.
env:
MATRIX_EC: ${{ inputs.matrix_ec }}
run: |
set -euo pipefail
ARGS=(--matrix -y --package-url "${{ inputs.package_url || env.RUSTFS_NIGHTLY_PACKAGE_URL }}" --log-file "${LOG_FILE}" --results-file "${RESULTS_FILE}")
if [ -n "${{ inputs.matrix_topo }}" ]; then
# Trailing value form: consumed as `--matrix-topo <topo> [ec]`.
ARGS+=(--matrix-topo "${{ inputs.matrix_topo }}")
if [ -n "${MATRIX_EC}" ]; then
ARGS+=("${MATRIX_EC}")
fi
fi
./auto-testing/rustfs-fault-tolerance-test.sh "${ARGS[@]}"
- name: Generate report
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
run: |
set -euo pipefail
{
echo "# RustFS fault-tolerance matrix report"
echo ""
echo "- Run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Package: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
echo "- auto-testing: \`${AUTO_TESTING_SHA}\`"
echo "- Topology filter: ${{ inputs.matrix_topo || 'all (10 topologies)' }}"
echo "- EC filter: ${{ inputs.matrix_ec || 'all levels' }}"
echo ""
echo "## Per-probe results"
echo ""
echo '```'
grep -E '^FT-(CASE|COMPLETE|SUMMARY|REPORT)' "${LOG_FILE}" || echo "(no FT-CASE lines found)"
echo '```'
echo ""
echo "## Health snapshots"
echo ""
for f in "${FUNCTIONAL_ARTIFACTS_DIR}"/evidence/*.code; do
[ -e "${f}" ] || continue
printf '%s -> %s\n' "$(basename "${f}" .code)" "$(cat "${f}")"
done
} > "${REPORT_FILE}"
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
# An unfiltered sweep must land exactly 300 cases (36 topology x EC
# combinations at 2*(nodes+1) cases each); filtered sweeps are held
# to internal consistency. Missing/harness-error rows always fail.
EXPECTED_TOTAL=""
if [ -z "${{ inputs.matrix_topo }}" ] && [ -z "${MATRIX_EC}" ]; then
EXPECTED_TOTAL=300
fi
jq -e --arg testing_sha "${AUTO_TESTING_SHA}" --argjson expected_total "${EXPECTED_TOTAL:-null}" '
.schema_version == 1 and
.auto_testing_sha == $testing_sha and
.complete == true and
.counts.expected == ($expected_total // .counts.expected) and
.counts.expected == .counts.seen and
.counts.harness_errors == 0 and
(.expected_cases | length) == .counts.expected and
(.expected_cases | unique | length) == .counts.expected and
(.seen_cases | length) == .counts.seen and
(.missing_cases | length) == 0 and
(.harness_errors | length) == 0 and
(.cases | length) == .counts.seen and
([.cases[].case_id] | unique | length) == .counts.seen and
([.cases[].case_id] | sort) == (.expected_cases | sort) and
all(.cases[]; (.case_id | startswith("M-")) and (.verdict == "pass" or .verdict == "known-divergence" or .verdict == "UNEXPECTED")) and
.counts.unexpected == ([.cases[] | select(.verdict == "UNEXPECTED")] | length) and
.counts.known_divergence == ([.cases[] | select(.verdict == "known-divergence")] | length)
' "${RESULTS_FILE}" >/dev/null
- name: Upload test logs & evidence
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-fault-tolerance-matrix-${{ github.run_id }}-${{ github.run_attempt }}
path: |
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/report.md
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/results.json
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/suite.log
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/evidence/
if-no-files-found: warn
- name: Cleanup environment (after)
timeout-minutes: 5
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}" || true
@@ -4,13 +4,14 @@
# behavior under drive and node loss against the erasure-coding contract and
# snapshots health-endpoint responses at every tier.
#
# A single-node 4 drives (SNMD): hide 1/2/3 drives, restore
# A single-node 4 drives (SNMD): revoke access to 1/2/3 drives, restore
# B multi-node 4x1 (one drive per node): stop 1/2/3 nodes, restore
# C multi-node 4x4 (16 drives, EC:4): stop 1 node (read-quorum boundary),
# stop 2 nodes, restore
# C2 multi-node 4x4 with EC:8: 2 nodes down puts 8 drives online -- reads
# satisfy the EC read quorum while the lock majority is broken (the
# reported divergence window: reads 503 with lock_quorum_unavailable)
# D multi-node 4x1: stop 1/2 nodes, then restore
#
# Expectations come from product source (default_parity_count, erasure set
# sizing). By default a "reads refused although the read quorum is met"
@@ -32,6 +33,11 @@ on:
description: 'Direct .deb URL. Required unless the nightly default is wanted.'
required: false
type: string
auto_testing_ref:
description: 'auto-testing branch, tag, or SHA (defaults to main)'
required: false
default: main
type: string
strict:
description: 'Fail the suite when reads are refused despite a met read quorum'
type: boolean
@@ -52,7 +58,7 @@ on:
permissions:
contents: read
# The suite stops services and hides drive dirs on the shared fleet; only one
# The suite stops services and revokes drive access on the shared fleet; only one
# functional suite may touch the environment at a time.
concurrency:
group: rustfs-shared-functional-tests-v2
@@ -77,7 +83,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -92,6 +98,7 @@ jobs:
printf 'FUNCTIONAL_ARTIFACTS_DIR=%s\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'LOG_FILE=%s/suite.log\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'REPORT_FILE=%s/report.md\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'RESULTS_FILE=%s/results.json\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'EVIDENCE_DIR=%s/evidence\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
} >> "${GITHUB_ENV}"
@@ -106,19 +113,27 @@ jobs:
- name: Checkout auto-testing scripts
timeout-minutes: 5
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
ref: ${{ steps.chain.outputs.testing_sha || inputs.auto_testing_ref || 'main' }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
path: auto-testing
persist-credentials: false
- name: Record auto-testing revision
run: |
set -euo pipefail
AUTO_TESTING_SHA="$(git -C auto-testing rev-parse HEAD)"
printf 'AUTO_TESTING_SHA=%s\n' "${AUTO_TESTING_SHA}" >> "${GITHUB_ENV}"
echo "auto-testing revision: ${AUTO_TESTING_SHA}"
- name: Show environment
run: |
uname -a
jq --version
aws --version
echo "auto-testing: ${AUTO_TESTING_SHA}"
df -h /data | tail -1
- name: Cleanup environment (before)
@@ -134,14 +149,14 @@ jobs:
CHAIN_MANIFEST: ${{ inputs.chain_manifest }}
run: python3 scripts/prepare_functional_package.py prepare
- name: Run fault-tolerance scenarios (A, B, C, C2)
- name: Run fault-tolerance scenarios (A, B, C, C2, D, E)
timeout-minutes: 45
id: test
# Standalone case failures are reported by the backlog manager.
# Chain evidence separately requires complete passing cases.
continue-on-error: true
run: |
ARGS=(--all -y --package-url "${{ steps.chain_package.outputs.package_url || inputs.package_url || env.RUSTFS_NIGHTLY_PACKAGE_URL }}" --log-file "${LOG_FILE}")
ARGS=(--all -y --package-url "${{ steps.chain_package.outputs.package_url || inputs.package_url || env.RUSTFS_NIGHTLY_PACKAGE_URL }}" --log-file "${LOG_FILE}" --results-file "${RESULTS_FILE}")
if [ "${{ inputs.strict }}" = "true" ]; then
ARGS+=(--strict)
fi
@@ -158,12 +173,13 @@ jobs:
echo ""
echo "- Run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Package: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
echo "- auto-testing: \`${AUTO_TESTING_SHA}\`"
echo "- Strict mode: ${{ inputs.strict || 'false' }}"
echo ""
echo "## Per-probe results"
echo ""
echo '```'
grep -E '^FT-(CASE|SUMMARY|REPORT)' "${LOG_FILE}" || echo "(no FT-CASE lines found)"
grep -E '^FT-(CASE|COMPLETE|SUMMARY|REPORT)' "${LOG_FILE}" || echo "(no FT-CASE lines found)"
echo '```'
echo ""
echo "## Health snapshots"
@@ -174,19 +190,30 @@ jobs:
done
} > "${REPORT_FILE}"
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
# Product gate: failing FT cases keep the run green — the report above and
# the backlog issue manager carry the signal. Only harness/environment
# breakdowns (no case verdicts at all) turn the workflow red.
FT_CASES="$(grep -cE '^FT-CASE:' "${LOG_FILE}" 2>/dev/null || true)"
FT_FAILED="$(grep -cE '^FT-CASE: .* verdict=UNEXPECTED' "${LOG_FILE}" 2>/dev/null || true)"
FT_PASSED=$(( ${FT_CASES:-0} - ${FT_FAILED:-0} ))
HARNESS_OK=0
if [ "${{ steps.test.outcome }}" = "success" ]; then
HARNESS_OK=1
elif [ "${{ steps.test.outcome }}" = "failure" ] && [ "${FT_CASES}" -gt 0 ] && [ "${FT_PASSED}" -ge 1 ]; then
HARNESS_OK=1
fi
[ "${HARNESS_OK}" = "1" ]
printf 'complete=false\n' >> "${GITHUB_OUTPUT}"
jq -e --arg testing_sha "${AUTO_TESTING_SHA}" '
.schema_version == 1 and
.auto_testing_sha == $testing_sha and
.selected_scenarios == ["A", "B", "C", "C2", "D", "E"] and
.complete == true and
.counts.expected == 44 and
.counts.seen == 44 and
.counts.harness_errors == 0 and
(.expected_cases | length) == 44 and
(.expected_cases | unique | length) == 44 and
(.seen_cases | length) == 44 and
(.seen_cases | unique | length) == 44 and
(.seen_cases | sort) == (.expected_cases | sort) and
(.missing_cases | length) == 0 and
(.harness_errors | length) == 0 and
(.cases | length) == 44 and
([.cases[].case_id] | sort) == (.expected_cases | sort) and
([.cases[].case_id] | unique | length) == 44 and
all(.cases[]; .verdict == "pass" or .verdict == "known-divergence" or .verdict == "UNEXPECTED") and
.counts.unexpected == ([.cases[] | select(.verdict == "UNEXPECTED")] | length) and
.counts.known_divergence == ([.cases[] | select(.verdict == "known-divergence")] | length)
' "${RESULTS_FILE}" >/dev/null
printf 'complete=true\n' >> "${GITHUB_OUTPUT}"
- name: Manage backlog issues (dedup / label / auto-close)
timeout-minutes: 2
@@ -196,7 +223,7 @@ jobs:
# green run. Never acts on cancelled runs. Logic lives in
# auto-testing/scripts/issue_manager.py, which parses the
# FT-CASE verdict lines from the suite log.
if: ${{ always() && steps.evidence.outcome == 'success' }}
if: ${{ always() && steps.evidence.outcome == 'success' && steps.chain_report.outputs.complete == 'true' }}
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
@@ -267,11 +294,12 @@ jobs:
- name: Upload test logs & evidence
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}
path: |
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/report.md
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/results.json
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/suite.log
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/evidence/
if-no-files-found: warn
@@ -358,7 +386,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}/fault-tolerance.json
+12 -9
View File
@@ -42,25 +42,28 @@ concurrency:
jobs:
prepare:
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.event == 'schedule' }}
runs-on: ubuntu-latest
runs-on: smoke-testing
timeout-minutes: 10
outputs:
manifest: ${{ steps.candidate.outputs.manifest }}
performance_ready: ${{ steps.perf_probe.outputs.performance_ready }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Resolve published candidate
id: candidate
env:
GH_TOKEN: ${{ github.token }}
# The staleness probe in resolve_functional_candidate.py reads the
# private rustfs/auto-testing repo; github.token cannot, and a probe
# failure silently keeps a stale testing_sha pinned for the chain.
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
BUILD_RUN_ATTEMPT: ${{ inputs.build_run_attempt }}
CHAIN_OUTPUT: ${{ runner.temp }}/chain-candidate.json
run: python3 scripts/resolve_functional_candidate.py
- name: Retain candidate identity
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-candidate-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-candidate.json
@@ -185,16 +188,16 @@ jobs:
complete-chain:
needs: [prepare, upgrade, s3, kms, tier, storage, heal, pool, security, replication, fault-tolerance, table, performance]
if: ${{ always() && needs.prepare.result != 'skipped' }}
runs-on: ubuntu-latest
runs-on: smoke-testing
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download suite evidence
id: download
continue-on-error: true
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: functional-chain-*-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-evidence
@@ -210,7 +213,7 @@ jobs:
--output "${RUNNER_TEMP}/chain-report.json"
- name: Retain chain report regardless of test verdict
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-report-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -228,7 +231,7 @@ jobs:
--directory "${RUNNER_TEMP}/chain-evidence"
--output "${RUNNER_TEMP}/chain-complete.json"
- name: Upload complete-chain evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-complete-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-complete.json
+4 -4
View File
@@ -66,7 +66,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -93,7 +93,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -366,7 +366,7 @@ jobs:
- name: Upload test logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-heal-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -467,7 +467,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-heal-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-heal-${{ github.run_id }}-${{ github.run_attempt }}/heal.json
+4 -4
View File
@@ -59,7 +59,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -85,7 +85,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -315,7 +315,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-kms-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -414,7 +414,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-kms-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-kms-${{ github.run_id }}-${{ github.run_attempt }}/kms.json
+12 -9
View File
@@ -61,10 +61,13 @@ on:
permissions:
contents: read
# The default performance nodes overlap the other suites' remote VMs, even
# though the runner differs. Hold the shared lock through cleanup as well.
# Workflow-level so the lock is held through cleanup as well.
concurrency:
group: rustfs-shared-functional-tests-v2
# Own group: the performance suite runs on the pf-testing fleet - a fully
# separate environment from the smoke-testing fleet the functional suites
# use. No shared node, no shared runner: it must never queue behind them
# (or block them).
group: rustfs-performance-suite
cancel-in-progress: false
defaults:
@@ -87,10 +90,10 @@ env:
jobs:
runner-check:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 5
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Check performance runner before entering its queue
@@ -113,7 +116,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -141,7 +144,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -304,7 +307,7 @@ jobs:
- name: Upload test logs & results
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-perf-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -346,7 +349,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-performance-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-performance-${{ github.run_id }}-${{ github.run_attempt }}/performance.json
+40 -6
View File
@@ -7,6 +7,11 @@ on:
description: Verified candidate and chain attempt from the chain driver
type: string
required: true
pool_timeout_minutes:
description: Wall-clock budget (minutes) for the pool run step
type: string
required: false
default: '240'
workflow_dispatch:
inputs:
rustfs_version:
@@ -39,6 +44,10 @@ on:
description: 'Run the pool decommission step (3-pool topology only)'
type: boolean
default: true
pool_timeout_minutes:
description: 'Wall-clock budget (minutes) for the whole pool run: fill + rebalance wait + decommission'
required: false
default: '240'
cleanup_before:
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
type: boolean
@@ -82,7 +91,8 @@ jobs:
pool-expansion-test:
name: Pool expansion / decommission test
runs-on: smoke-testing
timeout-minutes: 60
# Leave setup and finalizer headroom beyond the default 240-minute test.
timeout-minutes: 360
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
env:
RUSTFS_POOL_ADMIN_ENDPOINT: ${{ secrets.RUSTFS_POOL_ADMIN_ENDPOINT || vars.RUSTFS_POOL_ADMIN_ENDPOINT || 'http://rustfs-node1:9000' }}
@@ -92,7 +102,7 @@ jobs:
RUSTFS_POOL_NODE_ENDPOINTS: ${{ secrets.RUSTFS_POOL_NODE_ENDPOINTS || vars.RUSTFS_POOL_NODE_ENDPOINTS || 'http://rustfs-node1:9000 http://rustfs-node2:9000 http://rustfs-node3:9000 http://rustfs-node4:9000' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -104,7 +114,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -251,7 +261,12 @@ jobs:
- name: Run pool expansion & decommission test
id: pool_test
timeout-minutes: 45
# The suite budgets up to 24h each for rebalance and decommission
# (REBALANCE_TIMEOUT / DECOMMISSION_TIMEOUT in rustfs_pool_expand.sh);
# the previous 45m step cap killed four consecutive runs mid-rebalance
# with steps 1-7 green. 240m covers the observed pace (rebalance 1/4
# in ~21m) plus one decommission pass; override per dispatch.
timeout-minutes: ${{ inputs.pool_timeout_minutes || 240 }}
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
@@ -568,6 +583,25 @@ jobs:
echo "pool test script reported a failure (step FAIL verdict or die() abort)" >&2
failed=1
fi
# A runner-level kill (step timeout) truncates the suite without a
# FAIL verdict or an [ERROR] line, so require every expected
# [POOL-STEP] marker to be present with PASS - missing markers mean
# the run was truncated, not green. This also closes the gap for
# standalone dispatches, where the chain driver's evidence
# validation does not run.
last_step=6
if [ "${{ inputs.pools || '3' }}" = "3" ]; then
last_step=8
if [ "${{ inputs.run_decommission != 'false' }}" = "true" ]; then
last_step=9
fi
fi
for n in $(seq 1 "${last_step}"); do
grep -q "^\[POOL-STEP\] ${n} .* PASS" "${POOL_ARTIFACT_DIR}/pool-test.log" 2>/dev/null || {
echo "pool test log is missing [POOL-STEP] ${n} PASS (expected through step ${last_step}); the run was truncated" >&2
failed=1
}
done
[ "${failed}" -eq 0 ] || exit 1
- name: Upload functional report to dashboard
@@ -656,7 +690,7 @@ jobs:
- name: Upload test logs
timeout-minutes: 2
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-pool-test-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/rustfs-pool-${{ github.run_id }}-${{ github.run_attempt }}
@@ -764,7 +798,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-pool-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-pool-${{ github.run_id }}-${{ github.run_attempt }}/pool.json
@@ -71,7 +71,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -97,7 +97,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -323,7 +323,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-replication-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -419,7 +419,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-replication-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-replication-${{ github.run_id }}-${{ github.run_attempt }}/replication.json
+4 -4
View File
@@ -47,7 +47,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -73,7 +73,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -292,7 +292,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-s3-compat-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -391,7 +391,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-s3-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-s3-${{ github.run_id }}-${{ github.run_attempt }}/s3.json
+6 -6
View File
@@ -87,7 +87,7 @@ jobs:
# the workspace root would wipe the auto-testing clone above (that is
# exactly how run 33934141181 lost rustfs-security-test.sh).
- name: Checkout repository (for the OIDC live gate script)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
path: rustfs-repo
@@ -106,10 +106,10 @@ jobs:
if: ${{ inputs.chain_manifest != '' }}
env:
CHAIN_MANIFEST: ${{ inputs.chain_manifest }}
run: python3 scripts/functional_chain_evidence.py consume
run: python3 rustfs-repo/scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -325,7 +325,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-security-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -389,13 +389,13 @@ jobs:
CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }}
CHAIN_REPORT_OUTCOME: ${{ steps.report.outcome }}
run: >-
python3 scripts/functional_chain_evidence.py record --suite security
python3 rustfs-repo/scripts/functional_chain_evidence.py record --suite security
--report "${SECURITY_ARTIFACTS_DIR}/suite-report.md"
--output "${RUNNER_TEMP}/chain-security-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/security.json"
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-security-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-security-${{ github.run_id }}-${{ github.run_attempt }}/security.json
+4 -4
View File
@@ -56,7 +56,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -82,7 +82,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -307,7 +307,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-storage-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -406,7 +406,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-storage-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-storage-${{ github.run_id }}-${{ github.run_attempt }}/storage.json
+5 -5
View File
@@ -46,7 +46,7 @@ jobs:
timeout-minutes: 60
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -73,7 +73,7 @@ jobs:
- name: Checkout auto-testing scripts
timeout-minutes: 5
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -133,7 +133,7 @@ jobs:
if [ -n "${PACKAGE_URL}" ]; then
ARGS+=(--package-url "${PACKAGE_URL}")
elif [ -n "${RUSTFS_VERSION}" ] && [ "${RUSTFS_VERSION}" != "null" ]; then
ARGS+=(--package-url "https://github.com/rustfs/rustfs/releases/download/${RUSTFS_VERSION}/rustfs_${RUSTFS_VERSION}_amd64.deb")
ARGS+=(--package-url "https://github.com/rustfs/rustfs/releases/download/${RUSTFS_VERSION}/rustfs-linux-x86_64-gnu-v${RUSTFS_VERSION}.deb")
else
ARGS+=(--package-url "${RUSTFS_NIGHTLY_PACKAGE_URL}")
fi
@@ -281,7 +281,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-table-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -353,7 +353,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-table-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-table-${{ github.run_id }}-${{ github.run_attempt }}/table.json
+4 -4
View File
@@ -71,7 +71,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -95,7 +95,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -460,7 +460,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-tier-test-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ env.TIER_ARTIFACTS_DIR }}/
@@ -734,7 +734,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-tier-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-tier-${{ github.run_id }}-${{ github.run_attempt }}/tier.json
+5 -5
View File
@@ -89,7 +89,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -115,7 +115,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -207,7 +207,7 @@ jobs:
local version="$1" tag asset url
[ -n "${version}" ] && [ "${version}" != "null" ] || return 0
tag="${version#v}"
asset="rustfs_${tag//-/.}_amd64.deb"
asset="rustfs-linux-x86_64-gnu-v${tag}.deb"
url="https://github.com/rustfs/rustfs/releases/download/${tag}/${asset}"
if ! gh api "repos/rustfs/rustfs/releases/tags/${tag}" --jq '.assets[].name' 2>/dev/null | grep -qxF "${asset}"; then
echo "ERROR: release ${tag} has no downloadable asset ${asset}:" >&2
@@ -400,7 +400,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-upgrade-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -503,7 +503,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-upgrade-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-upgrade-${{ github.run_id }}-${{ github.run_attempt }}/upgrade.json
@@ -44,7 +44,7 @@ permissions:
jobs:
forced-failure:
name: Forced failure
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 5
steps:
- name: Fail on purpose
@@ -64,7 +64,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
@@ -29,6 +29,9 @@ concurrency:
jobs:
check-freshness:
name: Check scheduled validation freshness
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
# scheduled validations it watches stall.
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
@@ -36,7 +39,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Check latest scheduled runs
@@ -42,6 +42,9 @@ jobs:
github.event.workflow_run.event == 'schedule' &&
github.event.workflow_run.conclusion != 'success' &&
github.event.workflow_run.conclusion != 'failure'
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
# scheduled validations it watches stall.
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
@@ -49,7 +52,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update incomplete-run issue
+2 -2
View File
@@ -26,10 +26,10 @@ on:
jobs:
stale:
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
steps:
- uses: actions/stale@5bef64f19d7facfb25b37b414482c7164d639639 # v9
- uses: actions/stale@4391f3da665fdf50b6810c1a66712fb9ba21aa93 # v11.0.0
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
stale-issue-message: 'This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.'
+1 -1
View File
@@ -14,7 +14,7 @@ concurrency:
jobs:
update:
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- uses: overtrue/repo-visuals-action@ee2c632f6ce617e851fb46ea935ee8af762ebb93 # v1.4.0
+4 -4
View File
@@ -41,7 +41,7 @@ env:
jobs:
targets-live:
name: PostgreSQL, MySQL, AMQP, and NATS
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
@@ -52,7 +52,7 @@ jobs:
RUSTFS_TEST_NATS_URL: nats://127.0.0.1:4222
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -166,7 +166,7 @@ jobs:
- name: Upload target integration diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: targets-integration-${{ github.run_number }}-${{ github.run_attempt }}
path: artifacts/targets-live
@@ -183,7 +183,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+1 -1
View File
@@ -56,7 +56,7 @@ jobs:
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
+1
View File
@@ -25,6 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Presigned URLs honour only signed headers** (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an `x-amz-*` request header not listed in `X-Amz-SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned `PutObject` URL could add unsigned `x-amz-tagging`, `x-amz-storage-class`, `x-amz-website-redirect-location`, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in `SignedHeaders`; `x-amz-cf-id` (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged.
### Fixed
- **Two admin handlers reported a missing policy or user as `500 InternalError`**: `GET /rustfs/admin/v3/info-canned-policy` and `DELETE /rustfs/admin/v3/remove-user` wrapped every IAM error as an internal fault instead of classifying it, so a lookup for something that does not exist answered `500 InternalError` where the rest of the admin API answers `404 NoSuchResource` — `GET /rustfs/admin/v3/user-info` already did, and `iam_error_to_s3_error` already mapped `NoSuchPolicy`, `NoSuchUser`, `NoSuchServiceAccount` and `NoSuchTempAccount` to `NoSuchResource`. A client could not tell "does not exist" from a real server failure, and a controller that checks existence before creating (a Kubernetes operator reconciling policies or users) could never create the object: the missing-policy lookup failed with a `500` and it retried forever. Both handlers now map their IAM errors through the shared `iam_error_to_s3_error` helper, so a missing policy or user is `404 NoSuchResource` and the `error` is preserved as the S3 error's source; the three IAM calls in the removal path (temporary-user lookup, service-account lookup, delete) are all reclassified, and every other IAM error stays `500 InternalError`. Fixes #8117.
- **Vault backends accepted key names that escape the key prefix**: only the Local backend refused a key identifier containing a path separator, so on Vault KV2 `POST /rustfs/admin/v3/kms/keys` with a name such as `bad/name` succeeded and created a nested KV2 path that the key listing then reported as a directory rather than a key, and an identifier containing `..` addressed a record outside the configured key prefix once the HTTP client normalised the URL. Both Vault backends now apply the Local backend's containment rule at the point where the identifier becomes a path or a Transit key name: an empty identifier, one containing `/`, `\\` or NUL, or the dot segments `.` and `..` is refused with `400` (`InvalidKey`) before any request reaches Vault. The AWS backend is unaffected, since it addresses keys by ARN and alias. A Vault deployment that created such a key on an earlier build must re-create it under a plain name; objects encrypted under the old name stay readable only until that key is refused, so migrate them first. Refs rustfs/backlog#2474.
- **Fresh multi-pool bootstrap with distinct format creators**: a new deployment whose pools have their first endpoint on different nodes (for example two single-node pools) could never publish its initial `pool.bin`: each node held fresh-bootstrap proof only for the pool it formatted, the deployment-wide proof collapsed to none, and every node died with `pool metadata recovery required: no durable bootstrap identity or pool.bin replica is available` after the startup retry budget. The first pool's creator now mints the pending cluster identity on its own pool, every other creator copies that nonce-bound identity onto the pool it formatted first-hand, and the elected writer publishes `pool.bin` once every pool replica carries the same pending identity. Corrupt or disagreeing replicas, pools that merely have a format, expansion pools joining an initialized deployment, and restarts without first-hand proof still fail closed. Non-elected nodes that start before `pool.bin` exists, and the elected writer while it waits for the other creators, no longer latch their pool-metadata write gate for the life of the process. Refs rustfs/backlog#2338, rustfs/backlog#2375.
- **Lock RPC timeout storms** (#7363): the remote lock client no longer evicts and re-dials the shared internode HTTP/2 channel on every request deadline. A timeout evicts only when the peer has not completed any lock RPC for two deadlines, evictions and transport-failure re-dials are rate limited per peer (`RUSTFS_OBJECT_LOCK_RPC_EVICTION_COOLDOWN_MS`, default 5 s), and a timed-out request is left running instead of being reset (bounded per peer by `RUSTFS_OBJECT_LOCK_RPC_DETACHED_LIMIT`, default 256), so a slow lock endpoint can no longer drive the `RST_STREAM`/`GOAWAY too_many_resets`/reconnect loop. A lock granted after its caller timed out is released immediately, and unlocks that fail the quick retries continue on a deferred 1/2/4/8/16 s schedule before the server lease reclaims them. New `rustfs_remote_lock_*` metrics cover timeouts, evictions, suppressed evictions, detached streams, late completions and late releases per peer. Operator guide at `docs/operations/lock-rpc-storm-protection.md`.
Generated
+442 -454
View File
File diff suppressed because it is too large Load Diff
+42 -41
View File
@@ -20,6 +20,7 @@ members = [
"crates/common", # Shared utilities and data structures
"crates/data-usage", # Shared data usage models and algorithms
"crates/config", # Configuration management
"crates/connect-symbol-catalog", # Release-only CPU profile symbol catalog generator
"crates/credentials", # Credential management system
"crates/crypto", # Cryptography and security features
"crates/ecstore", # Erasure coding storage implementation
@@ -145,8 +146,7 @@ rustfs-zip = { path = "./crates/zip", version = "1.0.1" }
async-channel = "2.5.0"
async_zip = { default-features = false, version = "0.0.19" }
mysql_async = { default-features = false, version = "0.37.1" }
async-compression = { version = "0.4.47" }
async-recursion = "1.1.1"
async-compression = { version = "0.4.50" }
async-trait = "0.1.92"
async-nats = { version = "0.50.0", default-features = false }
axum = "0.8.9"
@@ -156,10 +156,10 @@ futures-lite = "2.6.1"
futures-util = "0.3.34"
pollster = "1.0.1"
pulsar = { default-features = false, version = "6.9.0" }
lapin = { default-features = false, version = "4.11.0" }
lapin = { default-features = false, version = "4.12.0" }
hyper = { version = "1.11.1" }
hyper-rustls = { default-features = false, version = "0.27.9" }
hyper-util = { version = "0.1.20" }
hyper-rustls = { default-features = false, version = "0.27.10" }
hyper-util = { version = "0.1.21" }
http = "1.5.0"
http-body = "1.1.0"
http-body-util = "0.1.5"
@@ -169,9 +169,9 @@ reqwest = "0.13.5"
rustfs-kafka-async = { version = "1.3.1" }
socket2 = { version = "0.6.5" }
tokio = { version = "1.53.1" }
tokio-rustls = { default-features = false, version = "0.26.5" }
tokio-rustls = { default-features = false, version = "0.26.6" }
tokio-stream = { version = "0.1.19" }
tokio-test = "0.4.5"
tokio-test = "0.4.6"
tokio-util = { version = "0.7.19" }
tonic = { version = "0.14.6" }
tonic-prost = { version = "0.14.6" }
@@ -244,13 +244,13 @@ atoi = "3.1.0"
atomic_enum = "0.3.0"
aws-config = { version = "1.12.0" }
aws-credential-types = { version = "1.3.0" }
aws-sdk-kms = { default-features = false, version = "1.118.0" }
aws-sdk-s3 = { default-features = false, version = "1.146.1" }
aws-sdk-sts = { default-features = false, version = "1.114.0" }
aws-sdk-kms = { default-features = false, version = "1.122.0" }
aws-sdk-s3 = { default-features = false, version = "1.151.0" }
aws-sdk-sts = { default-features = false, version = "1.118.0" }
aws-smithy-async = { version = "1.3.0" }
aws-smithy-http-client = { default-features = false, version = "1.4.2" }
aws-smithy-runtime-api = { version = "1.16.2" }
aws-smithy-types = { version = "1.6.4" }
aws-smithy-http-client = { default-features = false, version = "1.5.0" }
aws-smithy-runtime-api = { version = "1.19.0" }
aws-smithy-types = { version = "1.8.1" }
base64-simd = "0.8.0"
brotli = "9.0.0"
clap = { version = "4.6.7" }
@@ -265,11 +265,11 @@ datafusion = { default-features = false, version = "55.1.0" }
derive_builder = "0.20.2"
ed25519-dalek = "3.0.0"
enumset = "1.1.14"
faster-hex = "0.10.0"
faster-hex = "1.0.0"
flate2 = "1.1.10"
glob = "0.3.4"
google-cloud-storage = "1.18.0"
google-cloud-auth = "1.16.0"
google-cloud-storage = "1.20.0"
google-cloud-auth = "1.17.0"
hashbrown = { version = "0.17.1" }
# Base32 for RFC 6238 TOTP shared secrets (RFC 4648 unpadded, the alphabet
# every authenticator app expects). Already in the graph transitively.
@@ -278,7 +278,7 @@ hex-simd = "0.8.0"
highway = { version = "1.3.0" }
hostname = "0.4.2"
ipnetwork = { version = "0.21.1" }
lazy_static = "1.5.0"
lazy_static = "1.5.1"
libc = "0.2.189"
libsystemd = "0.7.2"
local-ip-address = "0.6.13"
@@ -292,6 +292,7 @@ moka = { version = "0.12.16" }
netif = "0.1.6"
num_cpus = { version = "1.17.0" }
nvml-wrapper = "0.13.0"
object = { version = "=0.40.0", default-features = false }
parking_lot = "0.12.5"
path-absolutize = "4.0.1"
percent-encoding = "2.3.2"
@@ -301,24 +302,24 @@ percent-encoding = "2.3.2"
qrcode-rs = { version = "2.1.1", default-features = false, features = ["std", "svg"] }
pin-project-lite = "0.2.17"
pretty_assertions = "1.4.1"
rand = { version = "0.10.2" }
rand = { version = "0.10.3" }
ratelimit = "2.0.0"
rayon = "1.12.0"
rustfs-erasure-codec = { version = "9.0.0" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.34.0" }
redis = { version = "1.7.0" }
redis = { version = "1.7.1" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.4" }
rustix = { version = "1.1.5" }
rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
s3s = { version = "0.16.0", features = ["minio"] }
s3s-sigv4 = { version = "0.16.0" }
s3s = { version = "0.17.0", features = ["minio"] }
s3s-sigv4 = "0.17.0"
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.3"
smallvec = { version = "1.16.1" }
siphasher = "1.0.4"
smallvec = { version = "1.16.2" }
compact_str = "0.10.0"
snap = "1.1.2"
starshard = { version = "2.3.0" }
@@ -326,13 +327,13 @@ strum = { version = "0.28.0" }
sysinfo = "0.39.6"
temp-env = "0.3.6"
tempfile = "3.27.0"
test-case = "3.3.1"
thiserror = "2.0.20"
test-case = "3.4.0"
thiserror = "2.0.21"
tracing = { version = "0.1.44" }
tracing-appender = "0.2.5"
tracing-core = "0.1.36"
tracing-error = "0.2.1"
tracing-opentelemetry = { version = "0.33" }
tracing-opentelemetry = { version = "0.34" }
tracing-subscriber = { version = "0.3.23" }
transform-stream = "0.3.1"
url = "2.5.8"
@@ -344,38 +345,38 @@ walkdir = "2.5.0"
winapi-util = "0.1.11"
windows = { version = "0.62.2" }
windows-sys = "0.61.2"
xxhash-rust = { version = "0.8.18" }
xxhash-rust = { version = "0.8.19" }
zip = "8.6.0"
zstd = "0.14.0"
# Observability and Metrics
metrics = "0.24.6"
metrics-util = "0.20"
dial9 = { version = "0.5.0", default-features = false }
opentelemetry = { version = "0.32.0" }
opentelemetry-appender-tracing = { version = "0.32.0" }
opentelemetry-otlp = { version = "0.32.0" }
opentelemetry-proto = { version = "0.32.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
opentelemetry_sdk = { version = "0.32.1" }
opentelemetry-semantic-conventions = { version = "0.32.1" }
opentelemetry-stdout = { version = "0.32.0" }
dial9 = { version = "0.5.2", default-features = false }
opentelemetry = { version = "0.33.0" }
opentelemetry-appender-tracing = { version = "0.33.0" }
opentelemetry-otlp = { version = "0.33.0" }
opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
opentelemetry_sdk = { version = "0.33.0" }
opentelemetry-semantic-conventions = { version = "0.33.0" }
opentelemetry-stdout = { version = "0.33.0" }
pyroscope = { version = "2.1.1" }
# FTP and SFTP
libunftp = { version = "0.23.0" }
libunftp = { version = "0.23.1" }
unftp-core = "0.1.0"
suppaftp = { version = "12.0.0" }
suppaftp = { version = "12.1.1" }
rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.63.3" }
russh-sftp = "3.0.0"
russh-sftp = "3.0.1"
# WebDAV
dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.5.4" }
rustfs-mimalloc = { version = "0.5.6" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.25.1", default-features = false }
hotpath = { version = "0.28.0", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
+3 -1
View File
@@ -106,7 +106,9 @@ RUN chmod +x /usr/bin/rustfs /entrypoint.sh
RUN addgroup -g 10001 -S rustfs && \
adduser -u 10001 -G rustfs -S rustfs -D && \
mkdir -p /data /logs && \
mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs && \
ln -s /data/.rustfs/events /opt/rustfs/events && \
ln -s /data/.rustfs/audit /opt/rustfs/audit && \
chown -R rustfs:rustfs /data /logs && \
chmod 0750 /data /logs
+3 -1
View File
@@ -44,7 +44,9 @@ RUN set -eux; \
WORKDIR /app
RUN set -eux; \
mkdir -p /data /logs; \
mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs; \
ln -s /data/.rustfs/events /opt/rustfs/events; \
ln -s /data/.rustfs/audit /opt/rustfs/audit; \
chown -R rustfs:rustfs /data /logs /app; \
chmod 0750 /data /logs
+3 -1
View File
@@ -110,7 +110,9 @@ RUN chmod +x /usr/bin/rustfs /entrypoint.sh
RUN groupadd -g 10001 rustfs && \
useradd -u 10001 -g rustfs -m -s /sbin/nologin rustfs && \
mkdir -p /data /logs && \
mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs && \
ln -s /data/.rustfs/events /opt/rustfs/events && \
ln -s /data/.rustfs/audit /opt/rustfs/audit && \
chown -R rustfs:rustfs /data /logs && \
chmod 0750 /data /logs
+3 -1
View File
@@ -240,7 +240,9 @@ WORKDIR /app
# Prepare data/log directories with sane defaults
RUN set -eux; \
mkdir -p /data /logs; \
mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs; \
ln -s /data/.rustfs/events /opt/rustfs/events; \
ln -s /data/.rustfs/audit /opt/rustfs/audit; \
chown -R rustfs:rustfs /data /logs /app; \
chmod 0750 /data /logs
+1 -1
View File
@@ -186,7 +186,7 @@
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2024 Beijing Henghesha Technology Co., Ltd.
Copyright 2024-2026 RustFS, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
+2 -2
View File
@@ -141,7 +141,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -195,7 +195,7 @@ docker run -d --name rustfs -p 9000:9000 \
Notes:
- `RUSTFS_NOTIFY_ENABLE=true` enables the global notify module switch.
- For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
- If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
- If queue dir is omitted, the official image maps `/opt/rustfs/events` into the persistent `/data` volume so it is writable by the runtime user and pending events survive container recreation. Override `queue_dir` when using another deployment layout.
- `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
- Since `1.0.0-beta.11`, webhook endpoints on private or container networks
(`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are
+1 -1
View File
@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
+129 -6
View File
@@ -163,6 +163,15 @@ pub struct MrfDurableRepairAnchor {
pub bucket_incarnation_id: Uuid,
}
/// A completed deep check proved that a legacy object is present and healthy,
/// but the format has no independent identity commitment that can discharge a
/// durable partial-write responsibility. Keep the journal entry and pause
/// automatic retries until an operator has repaired the source of truth.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MrfUnverifiedLegacyEvent {
pub anchor: MrfDurableRepairAnchor,
}
impl MrfDurableRepairAnchor {
/// Build a dischargeable anchor only when the caller supplies the storage
/// incarnation and the original ingress lease. Legacy replay records lack
@@ -324,6 +333,9 @@ pub enum MrfDurableAdmissionError {
/// cancel repair.
pub struct MrfDurableSubmission {
pub intent: MrfIntent,
/// Bucket incarnation observed by the committed-write caller, when
/// available. Old journal records and legacy callers remain unbound.
pub source_bucket_incarnation_id: Option<Uuid>,
pub response: oneshot::Sender<Result<(), MrfDurableAdmissionError>>,
}
@@ -344,6 +356,19 @@ pub async fn persist_partial_write_intent(
version_id: Option<Uuid>,
scope: MrfScope,
) -> Result<(), MrfDurableAdmissionError> {
persist_partial_write_intent_with_incarnation(bucket, object, version_id, scope, None).await
}
pub async fn persist_partial_write_intent_with_incarnation(
bucket: &str,
object: &str,
version_id: Option<Uuid>,
scope: MrfScope,
source_bucket_incarnation_id: Option<Uuid>,
) -> Result<(), MrfDurableAdmissionError> {
if source_bucket_incarnation_id.is_some_and(|incarnation| incarnation.is_nil()) {
return Err(MrfDurableAdmissionError::InvalidIdentity);
}
let intent = MrfIntent {
bucket: Arc::from(bucket),
object: Arc::from(object),
@@ -355,7 +380,7 @@ pub async fn persist_partial_write_intent(
enqueued_at_ms: unix_now_ms(),
attempts: 0,
};
persist_durable_intent(intent).await
persist_durable_intent(intent, source_bucket_incarnation_id).await
}
pub async fn persist_delete_marker_purge_intent(
@@ -368,6 +393,7 @@ pub async fn persist_delete_marker_purge_intent(
if version_id.is_nil() {
return Err(MrfDurableAdmissionError::InvalidIdentity);
}
let source_bucket_incarnation_id = delete_marker_purge.bucket_incarnation_id;
let intent = MrfIntent {
bucket: Arc::from(bucket),
object: Arc::from(object),
@@ -379,17 +405,23 @@ pub async fn persist_delete_marker_purge_intent(
enqueued_at_ms: unix_now_ms(),
attempts: 0,
};
persist_durable_intent_unconditionally(intent).await
persist_durable_intent_unconditionally(intent, Some(source_bucket_incarnation_id)).await
}
async fn persist_durable_intent(intent: MrfIntent) -> Result<(), MrfDurableAdmissionError> {
async fn persist_durable_intent(
intent: MrfIntent,
source_bucket_incarnation_id: Option<Uuid>,
) -> Result<(), MrfDurableAdmissionError> {
if !mrf_delivery_enabled() {
return Err(MrfDurableAdmissionError::Disabled);
}
persist_durable_intent_unconditionally(intent).await
persist_durable_intent_unconditionally(intent, source_bucket_incarnation_id).await
}
async fn persist_durable_intent_unconditionally(mut intent: MrfIntent) -> Result<(), MrfDurableAdmissionError> {
async fn persist_durable_intent_unconditionally(
mut intent: MrfIntent,
source_bucket_incarnation_id: Option<Uuid>,
) -> Result<(), MrfDurableAdmissionError> {
if intent.bucket.is_empty()
|| intent.object.is_empty()
|| intent.bucket.len() > MRF_MAX_IDENTITY_COMPONENT
@@ -404,7 +436,11 @@ async fn persist_durable_intent_unconditionally(mut intent: MrfIntent) -> Result
}
let (response, receipt) = oneshot::channel();
sender
.try_send(MrfDurableSubmission { intent, response })
.try_send(MrfDurableSubmission {
intent,
source_bucket_incarnation_id,
response,
})
.map_err(|err| match err {
mpsc::error::TrySendError::Full(_) => MrfDurableAdmissionError::Full,
mpsc::error::TrySendError::Closed(_) => MrfDurableAdmissionError::Unavailable,
@@ -782,6 +818,8 @@ const MRF_REPAIRED_EVENT_CAP: usize = 4096;
static MRF_REPAIRED_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfRepairedEvent>>> = OnceLock::new();
static MRF_VERIFIED_REPAIR_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfVerifiedRepairEvent>>> =
OnceLock::new();
static MRF_UNVERIFIED_LEGACY_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfUnverifiedLegacyEvent>>> =
OnceLock::new();
/// Record a legacy notification for compatibility. This is not an
/// acknowledgement of storage verification or durable repair completion.
@@ -859,6 +897,44 @@ pub fn take_mrf_verified_repair_events_for(bucket: &str) -> Vec<MrfVerifiedRepai
taken
}
/// Record a bounded, exact-identity notice that a legacy durable partial-write
/// was checked but cannot be proven from its on-disk format. This is not a
/// repair proof and must never release the durable responsibility.
pub fn note_mrf_unverified_legacy(event: MrfUnverifiedLegacyEvent) {
let registry = MRF_UNVERIFIED_LEGACY_EVENTS.get_or_init(|| std::sync::Mutex::new(std::collections::VecDeque::new()));
let Ok(mut events) = registry.lock() else {
return;
};
if events.contains(&event) {
return;
}
if events.len() >= MRF_REPAIRED_EVENT_CAP {
events.pop_front();
}
events.push_back(event);
}
/// Take held-legacy notices for one bucket. The durable ledger still verifies
/// the full anchor before changing the journal entry to its held state.
pub fn take_mrf_unverified_legacy_events_for(bucket: &str) -> Vec<MrfUnverifiedLegacyEvent> {
let Some(registry) = MRF_UNVERIFIED_LEGACY_EVENTS.get() else {
return Vec::new();
};
let Ok(mut events) = registry.lock() else {
return Vec::new();
};
let mut taken = Vec::new();
events.retain(|event| {
if event.anchor.bucket.as_ref() == bucket {
taken.push(event.clone());
false
} else {
true
}
});
taken
}
#[cfg(test)]
mod tests {
use super::*;
@@ -1233,6 +1309,53 @@ mod tests {
assert!(take_mrf_verified_repair_events_for("verified-bucket-a").is_empty());
assert_eq!(take_mrf_verified_repair_events_for("verified-bucket-b").len(), 1);
}
#[test]
fn unverified_legacy_events_are_bucket_scoped_and_bounded() {
let bucket = Arc::<str>::from("unverified-legacy-event-bucket");
let _ = take_mrf_unverified_legacy_events_for(bucket.as_ref());
let event = MrfUnverifiedLegacyEvent {
anchor: MrfDurableRepairAnchor {
kind: MrfKind::PartialWrite,
bucket: bucket.clone(),
object: Arc::from("unverified-object"),
version_id: None,
scope: Some(MrfScope {
pool_index: 1,
set_index: 2,
}),
delete_marker_purge: None,
lease: MrfIngressLease::new(912),
bucket_incarnation_id: Uuid::new_v4(),
},
};
note_mrf_unverified_legacy(event.clone());
note_mrf_unverified_legacy(event.clone());
assert_eq!(take_mrf_unverified_legacy_events_for(bucket.as_ref()), vec![event]);
assert!(take_mrf_unverified_legacy_events_for(bucket.as_ref()).is_empty());
let cap_bucket = Arc::<str>::from("unverified-legacy-event-cap-bucket");
for index in 0..MRF_REPAIRED_EVENT_CAP + 4 {
note_mrf_unverified_legacy(MrfUnverifiedLegacyEvent {
anchor: MrfDurableRepairAnchor {
kind: MrfKind::PartialWrite,
bucket: cap_bucket.clone(),
object: Arc::from(format!("object-{index}")),
version_id: None,
scope: Some(MrfScope {
pool_index: 1,
set_index: 2,
}),
delete_marker_purge: None,
lease: MrfIngressLease::new(u64::try_from(index).expect("test index fits")),
bucket_incarnation_id: Uuid::new_v4(),
},
});
}
let events = take_mrf_unverified_legacy_events_for(cap_bucket.as_ref());
assert_eq!(events.len(), MRF_REPAIRED_EVENT_CAP);
assert_eq!(events.first().expect("bounded ring is non-empty").anchor.object.as_ref(), "object-4");
}
}
#[tokio::test]
async fn partial_write_durable_admission_waits_for_persistence_and_renews_lease() {
+2 -2
View File
@@ -96,11 +96,11 @@ pub struct WorkloadAdmissionSnapshot {
pub class: WorkloadClass,
/// Current admission state.
pub state: AdmissionState,
/// Active work count when the owner exposes it.
/// Active work count or allocated/reserved permit units, as defined by the owner.
pub active: Option<usize>,
/// Queued work count when the owner exposes it.
pub queued: Option<usize>,
/// Admission limit when the owner exposes it.
/// Admission limit in the same units as `active`, when the owner exposes it.
pub limit: Option<usize>,
/// Optional state reason for disabled, throttled, saturated, or unknown states.
pub reason: Option<String>,
+17 -14
View File
@@ -168,16 +168,19 @@ Scanner cycle budget controls:
## Foreground write admission environment variables
Large direct `PutObject` requests and multipart `UploadPart` requests share one
per-process permit pool that bounds how many bodies are ingested and written
concurrently. Small direct PUTs stay on the legacy path.
Large direct `PutObject` requests, multipart `UploadPart` requests, and
server-side multipart `UploadPartCopy` requests share one per-process permit
pool that bounds concurrent body ingest and storage writes. Small direct PUTs
stay on the legacy path.
- `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE`
- enables the default-on pool; `false` keeps only the soft request counter.
- default is `true`.
- `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT`
- permits in the pool; `0` derives half of `RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS`, clamped to `32`.
- default is `0` (32 permits at stock settings).
- a positive value is an exact request-count limit shared by all gated writes.
- default is `0`: derive half of `RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS`, clamped to `32`, as large-write slots. Each slot has eight units. A gated direct PUT or unknown-size multipart part uses eight units; known-size parts use one unit per 8 MiB, rounded up, with a minimum of one and maximum of eight.
- stock settings therefore share 256 units across at most 32 large/unknown writes or 256 parts of up to 8 MiB. Mixed sizes consume the same budget. Admitted known parts of up to 64 MiB total at most 2 GiB of declared payload; larger streaming writes and queued bodies are separate. This is not a bound on total process or kernel memory.
- admission snapshots report allocated or reserved units in `active` and the unit budget in `limit` under automatic sizing. Explicit and strict limits still report request-count permits.
- `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES`
- smallest direct `PutObject` that takes a permit; unknown-size requests always do.
- default is `33554432` (32 MiB).
@@ -188,13 +191,13 @@ concurrently. Small direct PUTs stay on the legacy path.
- smallest `UploadPart` that takes a permit; `0` gates every part.
- default is `0`.
- `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`
- how long an `UploadPart` waits in the bounded queue for a permit before returning S3 `SlowDown`; `0` rejects immediately when the pool is full.
- default is `10000`. Parts wait before body ingest, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing on a full pool.
- RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. Keep it with margin below the shortest such timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
- how long an `UploadPart` or `UploadPartCopy` waits in the bounded queue for a permit before returning S3 `SlowDown`; `0` rejects immediately when the pool is full.
- default is `10000`. Parts wait before body ingest or source-object reads, so SDK-default clients that send every part of an upload concurrently drain through the pool instead of failing on a full pool.
- RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. `UploadPartCopy` waits before taking bucket lifecycle locks or opening source readers, and uses the unknown-size weight because the authoritative copy length is available only after source metadata and range validation. Keep the wait with margin below the shortest client/proxy timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
- `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MAX_PENDING`
- maximum `UploadPart` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
- default is `0`, which derives 16 times the permit limit (512 at stock settings).
- each queued HTTP/1 part holds whatever unread body the client already pushed into the connection's kernel receive buffer (an HTTP/2 part holds up to its flow-control window in process memory), so this depth also bounds that memory. RustFS leaves the receive buffer to kernel autotuning (see `RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES` below), which keeps an unread connection at the kernel's initial size (128 KiB on current Linux).
- maximum `UploadPart` and `UploadPartCopy` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
- default is `0`, which derives 16 times the large-write slot count, or the explicit request-count limit (512 at stock settings). Automatic subdivision does not enlarge this queue.
- each queued HTTP/1 part holds whatever unread body the client already pushed into the connection's kernel receive buffer (an HTTP/2 part holds up to its flow-control window in process memory). Autotuned receive buffers can remain large on reused connections; queue depth does not imply a fixed per-connection memory cost.
- `RUSTFS_PUT_FOREGROUND_ADMISSION_ENABLE`, `RUSTFS_PUT_FOREGROUND_ADMISSION_LIMIT`, `RUSTFS_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`
- experimental strict gate that applies to every foreground write regardless of size and replaces the pool above when enabled.
- default is disabled; enabling it with limit `0` disables foreground write admission entirely.
@@ -203,9 +206,9 @@ concurrently. Small direct PUTs stay on the legacy path.
- `RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES`
- fixed `SO_RCVBUF` for the API listener, inherited by every accepted socket; `0` leaves the receive buffer to kernel autotuning.
- default is `0`. Earlier releases hard-coded 4 MiB, which Linux doubles to 8 MiB and which disables autotuning, so every connection whose body was not being read yet (a multipart part queued for a foreground write permit) could accumulate up to 8 MiB of unread body in kernel memory; at SDK-default multipart concurrency that was enough to push a node into TCP memory pressure.
- with autotuning the per-connection receive ceiling is the kernel's (`net.ipv4.tcp_rmem` max, 6 MiB on stock Linux) instead of the former fixed 8 MiB, so a single very high-bandwidth-delay connection may see a somewhat lower ceiling; raise `net.ipv4.tcp_rmem` first, and set this variable only on kernels without receive-buffer autotuning (illumos/Solaris) or where the sysctl cannot be changed.
- the send buffer stays fixed at 4 MiB because the stock Linux send autotuning ceiling (`net.ipv4.tcp_wmem` max, 4 MiB) is lower than a GB-level response stream needs.
- default is `0`. A fixed buffer disables receive autotuning. Linux doubles the requested value for socket-memory accounting, subject to kernel limits; that capacity is not a measurement of queued payload or allocated memory.
- with autotuning the receive ceiling is controlled by the kernel (`net.ipv4.tcp_rmem` on Linux); its defaults vary with kernel version and memory. A connection can retain a buffer enlarged by previous requests while its next request is queued. Tune the kernel ceiling first, and set this variable only on kernels without receive-buffer autotuning (illumos/Solaris) or where the sysctl cannot be changed.
- the send buffer remains fixed at 4 MiB; this setting only controls the receive side.
## Remote tier timeout environment variables
+23 -7
View File
@@ -364,11 +364,14 @@ const _: () = assert!(!DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE);
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE";
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: bool = true;
/// Maximum automatic foreground write requests admitted concurrently per process.
/// Explicit maximum foreground write requests admitted concurrently per process.
///
/// `0` derives a conservative default from the local disk-read scheduler cap,
/// currently clamped to protect the commit path without making ordinary high
/// throughput uploads single-file.
/// `0` derives up to 32 large-write slots from the local disk-read scheduler.
/// Each automatic slot has eight units: a gated direct PUT or unknown-size
/// part uses all eight, while a known-size part uses one unit per 8 MiB,
/// rounded up and capped at eight. Thus stock settings share one budget across
/// at most 32 large/unknown writes or 256 parts of up to 8 MiB. A positive
/// override retains request-count semantics for every gated write.
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT";
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: usize = 0;
@@ -384,7 +387,9 @@ pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 32 * 10
/// Multipart pressure is often many moderate-sized parts rather than one very
/// large request. The default gates every multipart part through the same permit
/// pool as large/unknown-size PutObject while keeping small direct PUTs on the
/// legacy path.
/// legacy path. Server-side UploadPartCopy requests use this same multipart
/// admission path with unknown-size weighting because source metadata and range
/// validation happen after admission.
pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: &str =
"RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES";
pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_MIN_SIZE_BYTES: usize = 0;
@@ -412,7 +417,9 @@ pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 250;
/// wait with a 10 s request deadline. The wait must leave margin under the
/// shortest of those, not merely fall below an SDK default, so the part
/// receives S3 `SlowDown`/503 for the client to retry instead of losing its
/// connection (issue #7385). `0` rejects immediately when the pool is full.
/// connection (issue #7385). Server-side UploadPartCopy waits in the same queue
/// before taking bucket lifecycle locks or opening source readers. `0` rejects
/// immediately when the pool is full.
pub const ENV_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str =
"RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 10_000;
@@ -424,7 +431,7 @@ pub const DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 10_0
// the wait past any client timeout.
const _: () = assert!(DEFAULT_PUT_MULTIPART_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS * 3 <= 30_000);
/// Maximum multipart UploadPart requests waiting for a foreground write permit per process.
/// Maximum multipart UploadPart or UploadPartCopy requests waiting for a foreground write permit per process.
///
/// Parts beyond this queue depth are rejected with S3 `SlowDown`/503 without
/// waiting, so a genuinely saturated node still fails fast instead of holding
@@ -663,6 +670,15 @@ pub const ENV_OBJECT_LOCK_RPC_DETACHED_LIMIT: &str = "RUSTFS_OBJECT_LOCK_RPC_DET
/// Default per-peer budget of detached (timed-out but still running) lock RPCs: 256.
pub const DEFAULT_OBJECT_LOCK_RPC_DETACHED_LIMIT: usize = 256;
/// Environment variable for the maximum number of in-flight lock acquisition
/// RPCs admitted to one peer. Requests beyond this bound fail fast as
/// retryable contention so a slow endpoint cannot accumulate an unbounded
/// queue while its channel remains healthy.
pub const ENV_OBJECT_LOCK_RPC_REQUEST_LIMIT: &str = "RUSTFS_OBJECT_LOCK_RPC_REQUEST_LIMIT";
/// Default per-peer in-flight lock acquisition RPC admission limit.
pub const DEFAULT_OBJECT_LOCK_RPC_REQUEST_LIMIT: usize = 128;
/// Environment variable to enable object namespace lock diagnostics.
///
/// When enabled, RustFS emits slow lock acquisition and long lock hold
+4 -7
View File
@@ -164,13 +164,10 @@ pub const DEFAULT_HTTP1_MAX_BUF_SIZE: usize = 64 * 1024; // 64 KB
/// autotuning.
///
/// A fixed `SO_RCVBUF` is inherited by every accepted socket and disables
/// receive-buffer autotuning, so a connection whose request body is not being
/// read yet (a multipart part queued for a foreground write permit) lets up to
/// the fixed size of unread body accumulate in kernel memory — Linux doubles
/// the requested value, so the former hard-coded 4 MiB held up to 8 MiB per
/// queued connection (issue #7385). Autotuning keeps an unread connection at
/// the kernel's initial size and grows only connections that are being
/// drained. Set this only on kernels without receive-buffer autotuning
/// receive-buffer autotuning. Queued multipart bodies can accumulate in kernel
/// memory with either policy: a reused autotuned connection may already have
/// a large buffer. Socket capacity is not the same as queued payload or actual
/// memory allocation. Set this only on kernels without receive-buffer autotuning
/// (illumos/Solaris) or on very high-bandwidth-delay links where the kernel's
/// autotuning ceiling (`net.ipv4.tcp_rmem` on Linux) is too low and cannot be
/// raised.
+14
View File
@@ -30,6 +30,9 @@ pub const ENV_OBS_ENDPOINT_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_TIMEOUT_M
pub const ENV_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS";
pub const ENV_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS";
pub const ENV_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS";
/// Absolute path to a PEM bundle whose certificates are added to the system
/// trust roots used by OTLP/HTTP exporters.
pub const ENV_OBS_TLS_CA_FILE: &str = "RUSTFS_OBS_TLS_CA_FILE";
pub const ENV_OBS_PROFILING_ENDPOINT: &str = "RUSTFS_OBS_PROFILING_ENDPOINT";
pub const ENV_OBS_USE_STDOUT: &str = "RUSTFS_OBS_USE_STDOUT";
pub const ENV_OBS_SAMPLE_RATIO: &str = "RUSTFS_OBS_SAMPLE_RATIO";
@@ -55,6 +58,13 @@ pub const ENV_OBS_LOG_FILENAME: &str = "RUSTFS_OBS_LOG_FILENAME";
pub const ENV_OBS_LOG_ROTATION_TIME: &str = "RUSTFS_OBS_LOG_ROTATION_TIME";
pub const ENV_OBS_LOG_KEEP_FILES: &str = "RUSTFS_OBS_LOG_KEEP_FILES";
/// Span lifecycle events emitted by the JSON log layers (`none`|`close`|`full`).
///
/// Defaults to `none`: a record per span lifecycle multiplies log volume on
/// object hot paths, and emitting the ancestor span list on every record
/// produced multi-megabyte lines that journald truncated (backlog#2642).
pub const ENV_OBS_LOG_SPAN_EVENTS: &str = "RUSTFS_OBS_LOG_SPAN_EVENTS";
/// Log cleanup related configurations
pub const ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES";
pub const ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES";
@@ -99,6 +109,8 @@ pub const DEFAULT_OBS_LOG_DRY_RUN: bool = false;
pub const DEFAULT_OBS_LOG_MATCH_MODE_PREFIX: &str = "prefix";
pub const DEFAULT_OBS_LOG_MATCH_MODE: &str = "suffix";
pub const DEFAULT_OBS_LOG_SPAN_EVENTS: &str = "none";
/// Default values for observability configuration
// ### Supported Environment Values
// - `production` - Secure file-only logging
@@ -141,6 +153,7 @@ mod tests {
assert_eq!(ENV_OBS_LOG_FILENAME, "RUSTFS_OBS_LOG_FILENAME");
assert_eq!(ENV_OBS_LOG_ROTATION_TIME, "RUSTFS_OBS_LOG_ROTATION_TIME");
assert_eq!(ENV_OBS_LOG_KEEP_FILES, "RUSTFS_OBS_LOG_KEEP_FILES");
assert_eq!(ENV_OBS_LOG_SPAN_EVENTS, "RUSTFS_OBS_LOG_SPAN_EVENTS");
assert_eq!(ENV_OBS_TRACES_EXPORT_ENABLED, "RUSTFS_OBS_TRACES_EXPORT_ENABLED");
assert_eq!(ENV_OBS_METRICS_EXPORT_ENABLED, "RUSTFS_OBS_METRICS_EXPORT_ENABLED");
assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
@@ -178,6 +191,7 @@ mod tests {
assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP, "gzip");
assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD, "zstd");
assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE_PREFIX, "prefix");
assert_eq!(DEFAULT_OBS_LOG_SPAN_EVENTS, "none");
assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE, "suffix");
assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, "zstd");
}
+22
View File
@@ -0,0 +1,22 @@
[package]
name = "rustfs-connect-symbol-catalog"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
publish = false
[dependencies]
hex-simd.workspace = true
object = { workspace = true, features = ["read", "std", "elf"] }
serde = { workspace = true, features = ["derive"] }
serde_json.workspace = true
sha2.workspace = true
# Match pyroscope's pprof-rs frame normalization exactly.
symbolic-demangle = { version = "=13.9.0", default-features = false, features = ["rust"] }
[dev-dependencies]
object = { workspace = true, features = ["write", "elf"] }
[lints]
workspace = true
+284
View File
@@ -0,0 +1,284 @@
// Copyright 2024 RustFS Team
// SPDX-License-Identifier: Apache-2.0
//! Release tooling only: derive reviewed CPU symbol candidates from the final ELF.
use object::{Object, ObjectSymbol};
use serde::Serialize;
use sha2::{Digest, Sha256};
use std::collections::BTreeSet;
use std::fs::{self, OpenOptions};
use std::io::{self, Read, Write};
use std::path::Path;
const MAX_BINARY_BYTES: u64 = 1_073_741_824;
const MAX_CATALOG_BYTES: usize = 32 * 1024 * 1024;
const MAX_SYMBOLS: usize = 200_000;
const NORMALIZATION: &str = "pyroscope-2.1.1-symbolic-demangle-13.9.0";
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
struct Catalog<'a> {
schema_version: u8,
normalization: &'static str,
source_commit: &'a str,
executable_sha256: String,
os_family: &'static str,
architecture: &'static str,
build_features: Vec<String>,
symbols: BTreeSet<String>,
}
fn invalid(message: &'static str) -> io::Error {
io::Error::new(io::ErrorKind::InvalidData, message)
}
fn symbol_name(raw: &[u8]) -> Option<String> {
// This is the same lossy UTF-8 conversion and demangler used by pprof-rs.
let name = symbolic_demangle::demangle(&String::from_utf8_lossy(raw)).into_owned();
if name.is_empty()
|| name.len() > 4096
|| matches!(name.as_str(), "Unknown" | "<unresolved>")
|| name.bytes().any(|b| b < 32 || b == 127)
{
return None;
}
Some(name)
}
fn catalog(binary: &[u8], source: &str, target: &str, features: Vec<String>) -> io::Result<Vec<u8>> {
if source.len() != 40 || !source.bytes().all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) {
return Err(invalid("source commit must be a lowercase full SHA"));
}
let expected_arch = match target {
"x86_64-unknown-linux-gnu" => object::Architecture::X86_64,
"aarch64-unknown-linux-gnu" => object::Architecture::Aarch64,
_ => return Err(invalid("only supported Linux GNU CPU profiling targets have catalogs")),
};
if features.is_empty()
|| features.len() > 64
|| !features.iter().any(|f| f == "pyroscope")
|| features.iter().any(|f| {
f.is_empty()
|| f.len() > 64
|| !f.as_bytes().first().is_some_and(u8::is_ascii_lowercase)
|| !f
.bytes()
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-' || b == b'_')
})
|| features.windows(2).any(|pair| pair[0] >= pair[1])
{
return Err(invalid("build features must be sorted, unique and include pyroscope"));
}
let file = object::File::parse(binary).map_err(|_| invalid("cannot parse executable"))?;
if file.format() != object::BinaryFormat::Elf
|| file.architecture() != expected_arch
|| !matches!(file.kind(), object::ObjectKind::Executable | object::ObjectKind::Dynamic)
{
return Err(invalid("the executable does not match the Linux target"));
}
if file.symbol_table().is_none() {
return Err(invalid("the executable is stripped; retain its function symbol table"));
}
let mut symbols = BTreeSet::new();
let mut symbol_bytes = 0;
for symbol in file.symbols().chain(file.dynamic_symbols()) {
if symbol.kind() != object::SymbolKind::Text || symbol.is_undefined() {
continue;
}
if let Some(name) = symbol.name_bytes().ok().and_then(symbol_name)
&& !symbols.contains(&name)
{
symbol_bytes += name.len() + 3;
if symbol_bytes > MAX_CATALOG_BYTES {
return Err(invalid("symbol catalog exceeds its byte limit"));
}
symbols.insert(name);
}
if symbols.len() > MAX_SYMBOLS {
return Err(invalid("symbol catalog exceeds its reviewed symbol limit"));
}
}
if symbols.is_empty() {
return Err(invalid("the executable has no usable function symbols"));
}
let catalog = Catalog {
schema_version: 1,
normalization: NORMALIZATION,
source_commit: source,
executable_sha256: hex_simd::encode_to_string(Sha256::digest(binary), hex_simd::AsciiCase::Lower),
os_family: "LINUX",
architecture: if expected_arch == object::Architecture::X86_64 {
"x86_64"
} else {
"aarch64"
},
build_features: features,
symbols,
};
let json = serde_json::to_vec(&catalog).map_err(io::Error::other)?;
if json.len() > MAX_CATALOG_BYTES {
return Err(invalid("symbol catalog exceeds its byte limit"));
}
Ok(json)
}
fn main() -> io::Result<()> {
let args: Vec<_> = std::env::args().skip(1).collect();
let [binary_path, source, target, features_json, output] = args.as_slice() else {
return Err(invalid(
"usage: rustfs-connect-symbol-catalog BINARY SOURCE_SHA TARGET FEATURES_JSON OUTPUT",
));
};
let input = fs::File::open(binary_path)?;
if !input.metadata()?.is_file() || input.metadata()?.len() > MAX_BINARY_BYTES {
return Err(invalid("input must be a bounded regular executable"));
}
let mut binary = Vec::new();
input.take(MAX_BINARY_BYTES + 1).read_to_end(&mut binary)?;
if u64::try_from(binary.len()).map_err(io::Error::other)? > MAX_BINARY_BYTES {
return Err(invalid("executable grew beyond its byte limit"));
}
let features = serde_json::from_str(features_json).map_err(io::Error::other)?;
let json = catalog(&binary, source, target, features)?;
// Never overwrite another build's reviewed artifact.
let mut file = OpenOptions::new().write(true).create_new(true).open(Path::new(output))?;
file.write_all(&json)?;
file.sync_all()?;
println!("{}", hex_simd::encode_to_string(Sha256::digest(&json), hex_simd::AsciiCase::Lower));
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
fn executable() -> Vec<u8> {
let mut object =
object::write::Object::new(object::BinaryFormat::Elf, object::Architecture::X86_64, object::Endianness::Little);
let section = object.section_id(object::write::StandardSection::Text);
object.append_section_data(section, &[0xc3], 1);
object.add_symbol(object::write::Symbol {
name: b"_ZN4test4main17h0123456789abcdefE".to_vec(),
value: 0,
size: 1,
kind: object::SymbolKind::Text,
scope: object::SymbolScope::Linkage,
weak: false,
section: object::write::SymbolSection::Section(section),
flags: object::SymbolFlags::None,
});
let mut bytes = object.write().expect("write ELF symbol fixture");
// The object writer emits ET_REL; this fixture models the final ET_EXEC header.
bytes[16..18].copy_from_slice(&2_u16.to_le_bytes());
bytes
}
fn bulk_executable(count: usize, suffix_len: usize) -> Vec<u8> {
let mut object =
object::write::Object::new(object::BinaryFormat::Elf, object::Architecture::X86_64, object::Endianness::Little);
let section = object.section_id(object::write::StandardSection::Text);
object.append_section_data(section, &[0xc3], 1);
let suffix = "x".repeat(suffix_len);
for index in 0..count {
object.add_symbol(object::write::Symbol {
name: format!("rustfs::profile::function_{index:06}_{suffix}").into_bytes(),
value: 0,
size: 1,
kind: object::SymbolKind::Text,
scope: object::SymbolScope::Linkage,
weak: false,
section: object::write::SymbolSection::Section(section),
flags: object::SymbolFlags::None,
});
}
let mut binary = object.write().expect("write bulk ELF fixture");
binary[16..18].copy_from_slice(&2_u16.to_le_bytes());
binary
}
#[test]
fn catalog_binds_the_exact_executable_and_normalizes_function_names() {
let binary = executable();
let encoded =
catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()]).expect("generate catalog");
let value: serde_json::Value = serde_json::from_slice(&encoded).expect("catalog JSON");
assert_eq!(value["symbols"], serde_json::json!(["test::main"]));
assert_eq!(
value["executableSha256"],
hex_simd::encode_to_string(Sha256::digest(&binary), hex_simd::AsciiCase::Lower)
);
assert_eq!(value["normalization"], NORMALIZATION);
assert!(catalog(&binary, &"a".repeat(40), "aarch64-unknown-linux-gnu", vec!["pyroscope".into()]).is_err());
assert!(catalog(&binary, "bad", "x86_64-unknown-linux-gnu", vec!["pyroscope".into()]).is_err());
assert!(catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into(); 2]).is_err());
}
#[test]
fn stripped_executables_and_wrong_abis_are_refused() {
let mut binary = executable();
// Removing section headers models a fully stripped executable without dynsym.
binary[40..48].fill(0);
binary[60..64].fill(0);
let error = catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()])
.expect_err("stripped binary must not yield an empty but trusted catalog");
assert!(error.to_string().contains("stripped"));
assert!(catalog(&executable(), &"a".repeat(40), "x86_64-unknown-linux-musl", vec!["pyroscope".into()]).is_err());
}
#[test]
fn symbol_hash_matches_the_connect_nonce_vector() {
let nonce: Vec<u8> = (0..32).collect();
let name = symbol_name(b"core::hint::black_box").expect("reviewed function name");
let mut digest = Sha256::new();
digest.update(b"rustfs-connect-cpu-symbol-v1\0");
digest.update(nonce);
digest.update(name.as_bytes());
assert_eq!(
hex_simd::encode_to_string(digest.finalize(), hex_simd::AsciiCase::Lower),
"5d5b472a0dd09124bfedf0bd243f88d5e61952b8da15d7463da019137320d77c"
);
}
#[test]
fn unresolved_and_non_function_material_cannot_become_catalog_entries() {
assert!(symbol_name(b"Unknown").is_none());
assert!(symbol_name(b"<unresolved>").is_none());
assert!(symbol_name(b"secret\nname").is_none());
assert!(symbol_name(&vec![b'x'; 4097]).is_none());
assert!(catalog(b"not ELF", &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()]).is_err());
}
#[test]
fn release_sized_catalog_retains_all_function_names() {
let binary = bulk_executable(110_000, 160);
let encoded = catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()])
.expect("retain complete bounded symbol catalog");
assert!(encoded.len() > 16 * 1024 * 1024);
assert!(encoded.len() <= MAX_CATALOG_BYTES);
let value: serde_json::Value = serde_json::from_slice(&encoded).expect("catalog JSON");
assert_eq!(value["symbols"].as_array().expect("symbol array").len(), 110_000);
}
#[test]
fn catalogs_over_either_resource_limit_are_refused() {
let source = "a".repeat(40);
let too_many = catalog(
&bulk_executable(MAX_SYMBOLS + 1, 0),
&source,
"x86_64-unknown-linux-gnu",
vec!["pyroscope".into()],
)
.expect_err("too many function names must be refused");
assert!(too_many.to_string().contains("reviewed symbol limit"));
let too_large = catalog(
&bulk_executable(10_000, 3_500),
&source,
"x86_64-unknown-linux-gnu",
vec!["pyroscope".into()],
)
.expect_err("oversized catalog must be refused");
assert!(too_large.to_string().contains("byte limit"));
}
}
+3 -3
View File
@@ -384,17 +384,17 @@ pub(crate) fn is_cluster_heal_coordination_unavailable(error: &(dyn std::error::
message.contains("500 Internal Server Error") && message.contains("cluster heal coordination unavailable")
}
/// Wait for the restarted cluster to admit the first root heal request.
/// Wait for the restarted cluster to admit a root heal request and return its response.
pub(crate) async fn start_root_heal_when_control_ready(
heal_url: &str,
heal_body: &str,
access_key: &str,
secret_key: &str,
) -> ChaosResult<()> {
) -> ChaosResult<String> {
let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(45);
loop {
match signed_admin_post(heal_url, Some(heal_body), access_key, secret_key).await {
Ok(_) => return Ok(()),
Ok(body) => return Ok(body),
Err(error) if is_cluster_heal_coordination_unavailable(error.as_ref()) && tokio::time::Instant::now() < deadline => {
tokio::time::sleep(std::time::Duration::from_millis(250)).await;
}
+84 -3
View File
@@ -22,6 +22,8 @@
//! - Common test constants and utilities
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
use aws_sdk_s3::operation::list_buckets::ListBucketsError;
use aws_sdk_s3::{Client, Config};
use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
use http::header::{CONTENT_TYPE, HOST};
@@ -61,6 +63,41 @@ const TEST_PORT_COUNTER_PATH: &str = "/tmp/rustfs_e2e_next_port";
const TEST_PORT_LOCK_DIR: &str = "/tmp/rustfs_e2e_port_allocator.lock";
const TEST_PORT_LOCK_STALE_AFTER: Duration = Duration::from_secs(30);
fn list_buckets_readiness_error(err: &SdkError<ListBucketsError>) -> String {
// SDK Display reports only the category. Do not expose raw response bodies or headers.
let service_code = err.as_service_error().and_then(|error| error.code()).filter(|code| {
matches!(
*code,
"ServiceUnavailable"
| "ServerNotInitialized"
| "InternalError"
| "AccessDenied"
| "InvalidAccessKeyId"
| "SignatureDoesNotMatch"
)
});
let transport = match err {
SdkError::DispatchFailure(failure) if failure.is_timeout() => Some("timeout"),
SdkError::DispatchFailure(failure) if failure.is_io() => Some("io"),
SdkError::DispatchFailure(failure) if failure.is_user() => Some("user"),
SdkError::DispatchFailure(failure) if failure.is_other() => Some("other"),
_ => None,
};
let mut source: Option<&(dyn std::error::Error + 'static)> = Some(err);
let mut io_error = None;
while let Some(error) = source {
if let Some(error) = error.downcast_ref::<std::io::Error>() {
io_error = Some((error.kind(), error.raw_os_error()));
break;
}
source = error.source();
}
format!(
"{err}; http_status={:?}; service_code={service_code:?}; transport={transport:?}; io_error={io_error:?}",
err.raw_response().map(|response| response.status().as_u16())
)
}
fn capture_log_path(log_dir: &Path, temp_dir: &str) -> Option<PathBuf> {
let temp_name = Path::new(temp_dir).file_name()?.to_string_lossy();
Some(log_dir.join(format!("{temp_name}.log")))
@@ -1575,8 +1612,9 @@ impl RustFSTestClusterEnvironment {
///
/// Verifies service availability by calling the S3 `list_buckets` API against the requested node,
/// retries up to 120 times with a 1-second interval between attempts.
async fn wait_for_node_service_ready(&self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
async fn wait_for_node_service_ready(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let client = self.create_s3_client(node_idx)?;
let mut last_error = None;
for attempt in 0..120 {
match client.list_buckets().send().await {
@@ -1584,13 +1622,28 @@ impl RustFSTestClusterEnvironment {
info!("Cluster node {} service ready after {} attempts", node_idx, attempt + 1);
return Ok(());
}
Err(_) => {
Err(err) => {
last_error = Some(err);
sleep(Duration::from_secs(1)).await;
}
}
}
Err(format!("Cluster node {} service failed to become ready", node_idx).into())
let last_error = last_error.as_ref().map(list_buckets_readiness_error);
// Observe liveness only after the existing readiness budget is exhausted, before cleanup.
let process_status = match self.nodes[node_idx].process.as_mut() {
Some(process) => match process.try_wait() {
Ok(Some(status)) => format!("exited ({status})"),
Ok(None) => "running".to_string(),
Err(error) => format!("unavailable (kind={:?}, os_code={:?})", error.kind(), error.raw_os_error()),
},
None => "not started".to_string(),
};
Err(format!(
"Cluster node {node_idx} service failed to become ready; last ListBuckets error={last_error:?}; process_status={process_status}; capture_log_path={:?}",
self.node_capture_log_paths[node_idx]
)
.into())
}
/// Create an S3 client configured to communicate with a specific cluster node.
@@ -2101,6 +2154,34 @@ mod tests {
verify_awscurl_path(&executable).expect("an available client with a working help command should pass");
}
#[test]
fn readiness_error_reports_transport_without_sensitive_source_text() {
use aws_sdk_s3::error::ConnectorError;
let sensitive = "request credentials must not appear in diagnostics";
for (connector, category) in [
(
ConnectorError::io(std::io::Error::new(ErrorKind::ConnectionReset, sensitive).into()),
"io",
),
(ConnectorError::timeout(sensitive.into()), "timeout"),
(ConnectorError::user(sensitive.into()), "user"),
(ConnectorError::other(sensitive.into(), None), "other"),
] {
let diagnostic = list_buckets_readiness_error(&SdkError::dispatch_failure(connector));
assert!(diagnostic.contains(&format!("transport=Some(\"{category}\")")), "{diagnostic}");
assert!(diagnostic.contains("http_status=None"), "{diagnostic}");
assert!(!diagnostic.contains(sensitive), "{diagnostic}");
if category == "io" {
assert!(diagnostic.contains("io_error=Some((ConnectionReset, None))"), "{diagnostic}");
}
}
let os_error = std::io::Error::from_raw_os_error(13);
let expected = format!("io_error=Some(({:?}, Some(13)))", os_error.kind());
let diagnostic = list_buckets_readiness_error(&SdkError::dispatch_failure(ConnectorError::io(os_error.into())));
assert!(diagnostic.contains(&expected), "{diagnostic}");
}
#[test]
fn capture_log_path_uses_temp_directory_basename() {
assert_eq!(
@@ -298,7 +298,11 @@ mod tests {
assert_eq!(empty_head_resp.content_disposition(), None);
assert_eq!(empty_head_resp.content_encoding(), None);
assert_eq!(empty_head_resp.content_language(), None);
assert_eq!(empty_head_resp.content_type(), None);
assert_eq!(
empty_head_resp.content_type(),
Some("binary/octet-stream"),
"REPLACE without an explicit content type must default to binary/octet-stream, matching S3"
);
assert_eq!(empty_head_resp.expires_string(), None);
assert_eq!(empty_head_resp.website_redirect_location(), None);
@@ -394,7 +398,15 @@ mod tests {
assert_eq!(head.content_disposition(), (field == "content-disposition").then_some("attachment"));
assert_eq!(head.content_encoding(), (field == "content-encoding").then_some("gzip"));
assert_eq!(head.content_language(), (field == "content-language").then_some("de"));
assert_eq!(head.content_type(), (field == "content-type").then_some("text/field"));
assert_eq!(
head.content_type(),
Some(if field == "content-type" {
"text/field"
} else {
"binary/octet-stream"
}),
"REPLACE without an explicit content type must default to binary/octet-stream, matching S3"
);
assert_eq!(
head.expires_string(),
(field == "expires").then_some(replacement_expires_http_date.as_str())
@@ -420,7 +432,11 @@ mod tests {
.send()
.await
.expect("HEAD failed for metadata collision case");
assert_eq!(collision_head.content_type(), None);
assert_eq!(
collision_head.content_type(),
Some("binary/octet-stream"),
"user-metadata under the reserved 'content-type' key must not satisfy the system content-type default"
);
assert_eq!(collision_head.content_encoding(), None);
assert_eq!(
collision_head.metadata().and_then(|metadata| metadata.get("content-type")),
@@ -282,7 +282,7 @@ async fn sdk_list_bucket_and_list_bucket_versions_permissions_are_independent()
}
#[tokio::test]
async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers() -> TestResult {
async fn explicit_force_delete_permission_removes_prefix_versions_and_delete_markers() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
@@ -300,7 +300,14 @@ async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers()
put(&root, bucket, "folder-sibling/keep.txt").await?;
let keep = versions(&root, bucket, "keep.txt").await?;
let sibling = versions(&root, bucket, "folder-sibling/").await?;
let user = policy_user(&env, "consoleAdmin", None).await?;
let user = policy_user(
&env,
"explicit-force-delete",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")}
]})),
)
.await?;
force_delete(&user, bucket, "folder/").await?;
assert!(
@@ -322,6 +329,126 @@ async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers()
Ok(())
}
#[tokio::test]
async fn force_delete_header_requires_explicit_permission() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let root = env.create_s3_client();
let admin = policy_user(&env, "consoleAdmin", None).await?;
let owner_bucket = "force-owner-bucket";
root.create_bucket().bucket(owner_bucket).send().await?;
put(&root, owner_bucket, "kept-until-force.txt").await?;
force_delete_bucket(&root, owner_bucket, "x-rustfs-force-delete").await?;
assert!(
head_bucket_missing(&root, owner_bucket).await?,
"root force-delete must remove a non-empty bucket"
);
let denied_bucket = "force-admin-denied";
root.create_bucket().bucket(denied_bucket).send().await?;
put(&root, denied_bucket, "folder/child.txt").await?;
put(&root, denied_bucket, "plain.txt").await?;
assert_boxed_denied(force_delete(&admin, denied_bucket, "folder/").await);
assert!(
!versions(&root, denied_bucket, "folder/").await?.is_empty(),
"consoleAdmin without s3:ForceDeleteObject must not recursively delete"
);
assert_boxed_denied(force_delete_bucket(&admin, denied_bucket, "x-rustfs-force-delete").await);
assert_boxed_denied(force_delete_bucket(&admin, denied_bucket, "x-minio-force-delete").await);
assert!(
!head_bucket_missing(&root, denied_bucket).await?,
"consoleAdmin without s3:ForceDeleteBucket must not force-delete a non-empty bucket"
);
let not_empty = admin.delete_bucket().bucket(denied_bucket).send().await;
let not_empty = not_empty.expect_err("plain delete of a non-empty bucket must fail");
assert_eq!(
not_empty.as_service_error().and_then(|error| error.code()),
Some("BucketNotEmpty"),
"plain delete must keep the emptiness check, got {not_empty:?}"
);
admin.delete_object().bucket(denied_bucket).key("plain.txt").send().await?;
assert!(
versions(&root, denied_bucket, "plain.txt").await?.is_empty(),
"plain DeleteObject without the force header must stay allowed for consoleAdmin"
);
let object_bucket = "force-explicit-object";
root.create_bucket().bucket(object_bucket).send().await?;
put(&root, object_bucket, "folder/child.txt").await?;
put(&root, object_bucket, "keep.txt").await?;
let keep = versions(&root, object_bucket, "keep.txt").await?;
let object_user = policy_user(
&env,
"explicit-object-force",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{object_bucket}/*")}
]})),
)
.await?;
force_delete(&object_user, object_bucket, "folder/").await?;
assert!(versions(&root, object_bucket, "folder/").await?.is_empty());
assert_eq!(versions(&root, object_bucket, "keep.txt").await?, keep);
let allowed_bucket = "force-explicit-bucket";
root.create_bucket().bucket(allowed_bucket).send().await?;
put(&root, allowed_bucket, "inside.txt").await?;
let bucket_user = policy_user(
&env,
"explicit-bucket-force",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteBucket","s3:ForceDeleteBucket"],"Resource":[
format!("arn:aws:s3:::{allowed_bucket}"), format!("arn:aws:s3:::{allowed_bucket}/*")
]}
]})),
)
.await?;
force_delete_bucket(&bucket_user, allowed_bucket, "x-minio-force-delete").await?;
assert!(
head_bucket_missing(&root, allowed_bucket).await?,
"an explicit s3:ForceDeleteBucket grant must remove a non-empty bucket"
);
Ok(())
}
async fn force_delete_bucket(
client: &Client,
bucket: &str,
header: &str,
) -> Result<(), Box<SdkError<aws_sdk_s3::operation::delete_bucket::DeleteBucketError>>> {
let header: &'static str = match header {
"x-rustfs-force-delete" => "x-rustfs-force-delete",
"x-minio-force-delete" => "x-minio-force-delete",
other => panic!("unexpected force-delete header {other}"),
};
client
.delete_bucket()
.bucket(bucket)
.customize()
.mutate_request(move |request| {
request.headers_mut().insert(header, "true");
})
.send()
.await
.map(|_| ())
.map_err(Box::new)
}
async fn head_bucket_missing(client: &Client, bucket: &str) -> TestResult<bool> {
match client.head_bucket().bucket(bucket).send().await {
Ok(_) => Ok(false),
Err(error) => {
let code = error.as_service_error().and_then(|service| service.code());
if matches!(code, Some("NotFound" | "NoSuchBucket")) {
Ok(true)
} else {
Err(error.into())
}
}
}
}
#[tokio::test]
async fn force_delete_authorizes_only_its_path_scope_without_list_permissions() -> TestResult {
init_logging();
@@ -339,7 +466,7 @@ async fn force_delete_authorizes_only_its_path_scope_without_list_permissions()
&env,
"delete-only",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":[
format!("arn:aws:s3:::{bucket}/selected.txt"), format!("arn:aws:s3:::{bucket}/selected.txt/*")
]},
{"Effect":"Deny","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/selected.txt-sibling")}
@@ -376,7 +503,7 @@ async fn force_directory_delete_cannot_remove_an_unauthorized_colliding_parent()
&env,
"parent-denier",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/collision.txt"),
"Condition":{"StringEquals":{"s3:VersionId":protected_parent_version}}}
]})),
@@ -409,7 +536,7 @@ async fn force_unversioned_directory_requires_only_delete_object() -> TestResult
&env,
"unversioned-deleter",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":"s3:DeleteObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Allow","Action":["s3:DeleteObject","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/*")}
]})),
)
@@ -438,7 +565,7 @@ async fn force_delete_denied_child_preserves_every_object_despite_bucket_allow()
&env,
"child-denier",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Deny","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/folder/z-denied.txt")}
]})),
)
@@ -468,7 +595,7 @@ async fn force_delete_denied_child_preserves_every_object_despite_bucket_allow()
&env,
"replica-deleter",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":"s3:DeleteObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Allow","Action":["s3:DeleteObject","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Allow","Action":"s3:ReplicateDelete","Resource":format!("arn:aws:s3:::{bucket}/*")}
]})),
)
@@ -500,7 +627,7 @@ async fn force_delete_denied_historical_version_preserves_versions_and_markers()
put(&root, bucket, "folder/a-allowed.txt").await?;
let policy = |version: &str| {
json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/folder/*"),
"Condition":{"StringEquals":{"s3:VersionId":version}}}
]})
@@ -628,7 +755,7 @@ async fn force_delete_checks_every_version_page_before_mutation() -> TestResult
.try_collect::<Vec<_>>()
.await?;
put(&root, bucket, "folder/z-denied.txt").await?;
let allow = json!({"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")});
let allow = json!({"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")});
let user = policy_user(
&env,
"paged-deleter",
+219 -25
View File
@@ -34,6 +34,9 @@ use crate::common::{
};
use crate::replication_extension_test::LOOPBACK_REPLICATION_TARGET_ENV;
use aws_sdk_s3::Client;
use aws_sdk_s3::config::retry::RetryConfig;
use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
use aws_sdk_s3::operation::{get_object::GetObjectError, put_object::PutObjectError};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use http::{Method, StatusCode};
@@ -41,7 +44,7 @@ use sha2::{Digest, Sha256};
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::time::Duration;
use tokio::time::{Instant, sleep};
use tokio::time::{Instant, sleep, timeout_at};
use uuid::Uuid;
pub(crate) type TestResult<T = ()> = Result<T, Box<dyn std::error::Error + Send + Sync>>;
@@ -400,9 +403,9 @@ pub(crate) async fn put_inventory(
Ok(inventory)
}
/// Retry only transport-level service availability failures while a data
/// movement operation changes the pool map. Generic InternalError responses
/// remain fatal because accepting them would hide server defects.
/// Retry only explicit S3 availability responses while a data movement
/// operation changes the pool map. Transport failures and InternalError
/// responses remain fatal because callers already require a ready endpoint.
pub(crate) async fn put_inventory_retrying(
client: &Client,
bucket: &str,
@@ -449,19 +452,48 @@ where
let deadline = Instant::now() + timeout;
let mut delay = Duration::from_millis(50);
loop {
let last_error = match probe().await {
Ok(true) => return Ok(()),
Ok(false) => format!("{label} still false"),
Err(error) => error.to_string(),
};
if Instant::now() >= deadline {
return Err(format!("{label} did not become true within {timeout:?}: {last_error}").into());
return Err(format!("{label} did not become true within {timeout:?}").into());
}
sleep(delay).await;
// A pending request must consume the same budget as unsuccessful probes.
match timeout_at(deadline, probe()).await {
Err(_) => return Err(format!("{label} did not become true within {timeout:?}").into()),
Ok(Err(error)) => return Err(error),
Ok(Ok(true)) => return Ok(()),
Ok(Ok(false)) => {}
}
tokio::time::sleep_until((Instant::now() + delay).min(deadline)).await;
delay = (delay * 2).min(Duration::from_secs(1));
}
}
// Boxed SDK errors retain the operation's service code; Display only says
// "service error" and cannot distinguish convergence from a server defect.
pub(crate) fn s3_probe_error_code<'a>(error: &'a (dyn std::error::Error + Send + Sync + 'static)) -> Option<&'a str> {
error
.downcast_ref::<SdkError<GetObjectError>>()
.and_then(|error| error.as_service_error())
.and_then(ProvideErrorMetadata::code)
.or_else(|| {
error
.downcast_ref::<SdkError<PutObjectError>>()
.and_then(|error| error.as_service_error())
.and_then(ProvideErrorMetadata::code)
})
}
pub(crate) fn s3_probe_client(client: &Client) -> Client {
// The probe owns the retry policy. SDK retries must not hide a response
// that the probe would classify as fatal.
Client::from_conf(
client
.config()
.to_builder()
.retry_config(RetryConfig::standard().with_max_attempts(1))
.build(),
)
}
pub(crate) async fn cluster_admin(
cluster: &RustFSTestClusterEnvironment,
method: Method,
@@ -598,15 +630,20 @@ pub(crate) async fn wait_for_replicated_bytes(
expected: &[u8],
timeout: Duration,
) -> TestResult {
let client = s3_probe_client(client);
wait_until(
timeout,
|| async {
match get_object_bytes(client, bucket, key).await {
match get_object_bytes(&client, bucket, key).await {
Ok(got) if got.as_slice() == expected => Ok(true),
Ok(_) => Ok(false),
Ok(got) => Err(format!(
"replicated object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
sha256_hex(expected),
sha256_hex(&got)
)
.into()),
Err(error) => {
let message = error.to_string();
if message.contains("NoSuchKey") || message.contains("NotFound") {
if matches!(s3_probe_error_code(error.as_ref()), Some("NoSuchKey" | "NotFound")) {
Ok(false)
} else {
Err(error)
@@ -988,6 +1025,7 @@ pub(crate) async fn list_pools_json(cluster: &RustFSTestClusterEnvironment) -> T
}
pub(crate) async fn retrying_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, timeout: Duration) -> TestResult {
let client = s3_probe_client(client);
wait_until(
timeout,
|| {
@@ -999,8 +1037,7 @@ pub(crate) async fn retrying_put(client: &Client, bucket: &str, key: &str, body:
match put_object(&client, &bucket, &key, body).await {
Ok(()) => Ok(true),
Err(error) => {
let message = error.to_string();
if message.contains("SlowDown") || message.contains("ServiceUnavailable") || message.contains("503") {
if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
Ok(false)
} else {
Err(error)
@@ -1021,19 +1058,20 @@ pub(crate) async fn retrying_get_equals(
expected: &[u8],
timeout: Duration,
) -> TestResult {
let client = s3_probe_client(client);
wait_until(
timeout,
|| async {
match get_object_bytes(client, bucket, key).await {
match get_object_bytes(&client, bucket, key).await {
Ok(got) if got.as_slice() == expected => Ok(true),
Ok(_) => Ok(false),
Ok(got) => Err(format!(
"object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
sha256_hex(expected),
sha256_hex(&got)
)
.into()),
Err(error) => {
let message = error.to_string();
if message.contains("NoSuchKey")
|| message.contains("SlowDown")
|| message.contains("ServiceUnavailable")
|| message.contains("503")
{
if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
Ok(false)
} else {
Err(error)
@@ -1046,6 +1084,162 @@ pub(crate) async fn retrying_get_equals(
.await
}
#[cfg(test)]
mod retry_tests {
use super::*;
use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction, Operation, SeedMetadata};
use std::cell::Cell;
fn client(target: &FakeS3Target) -> Client {
Client::from_conf(build_test_s3_config(
target.endpoint(),
FAKE_ACCESS_KEY,
FAKE_SECRET_KEY,
None,
"distributed-retry-test",
))
}
#[tokio::test]
async fn wait_until_preserves_first_error() {
let attempts = Cell::new(0);
let error = wait_until(
Duration::from_secs(5),
|| {
attempts.set(attempts.get() + 1);
std::future::ready(if attempts.get() == 1 {
Err(std::io::Error::new(std::io::ErrorKind::PermissionDenied, "permanent probe failure").into())
} else {
Ok(true)
})
},
"permanent failure",
)
.await
.expect_err("a later success must not hide a permanent failure");
assert_eq!(attempts.get(), 1);
assert_eq!(
error
.downcast_ref::<std::io::Error>()
.expect("preserve the original error")
.kind(),
std::io::ErrorKind::PermissionDenied
);
}
#[tokio::test]
async fn wait_until_retries_explicit_pending_state() -> TestResult {
let attempts = Cell::new(0);
wait_until(
Duration::from_secs(5),
|| {
attempts.set(attempts.get() + 1);
std::future::ready(Ok(attempts.get() == 2))
},
"eventual readiness",
)
.await?;
assert_eq!(attempts.get(), 2);
Ok(())
}
#[tokio::test]
async fn wait_until_bounds_a_pending_probe() {
let error = tokio::time::timeout(
Duration::from_secs(5),
wait_until(Duration::from_millis(10), std::future::pending, "hung probe"),
)
.await
.expect("the probe deadline must finish before the test watchdog")
.expect_err("a permanently pending probe must time out");
assert!(error.to_string().contains("hung probe did not become true"));
}
#[tokio::test]
async fn wait_until_does_not_start_a_probe_after_its_deadline() {
let attempts = Cell::new(0);
wait_until(
Duration::ZERO,
|| {
attempts.set(attempts.get() + 1);
std::future::ready(Ok(true))
},
"expired budget",
)
.await
.expect_err("an expired budget must not admit another probe");
assert_eq!(attempts.get(), 0);
}
#[tokio::test]
async fn retrying_put_does_not_hide_internal_error() -> TestResult {
let target = FakeS3Target::start().await?;
target.create_bucket("retry-probe");
target.inject(Operation::PutObject, FaultAction::ResponseStatus(500), 1);
let error = retrying_put(&client(&target), "retry-probe", "key", b"body".to_vec(), Duration::from_secs(5))
.await
.expect_err("InternalError must remain fatal even if the next PUT would succeed");
assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
assert_eq!(target.requests().len(), 1);
Ok(())
}
#[tokio::test]
async fn retrying_put_allows_transient_availability_errors() -> TestResult {
let target = FakeS3Target::start().await?;
target.create_bucket("retry-probe");
for status in [429, 503] {
target.inject(Operation::PutObject, FaultAction::ResponseStatus(status), 1);
retrying_put(&client(&target), "retry-probe", "key", b"body".to_vec(), Duration::from_secs(5)).await?;
assert_eq!(target.take_requests().len(), 2);
}
Ok(())
}
#[tokio::test]
async fn object_polls_reject_wrong_object_bytes() -> TestResult {
let target = FakeS3Target::start().await?;
target.create_bucket("retry-probe");
target.put_seed_object("retry-probe", "key", "wrong bytes", &SeedMetadata::default());
let error = retrying_get_equals(&client(&target), "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
.await
.expect_err("a successful response with wrong bytes must fail immediately");
assert!(error.to_string().contains("bytes mismatch"));
assert_eq!(target.take_requests().len(), 1);
let error = wait_for_replicated_bytes(&client(&target), "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
.await
.expect_err("replication may lag, but it must not return corrupt bytes");
assert!(error.to_string().contains("bytes mismatch"));
assert_eq!(target.requests().len(), 1);
Ok(())
}
#[tokio::test]
async fn object_polls_distinguish_committed_and_replicated_keys() -> TestResult {
let target = FakeS3Target::start().await?;
target.create_bucket("retry-probe");
target.put_seed_object("retry-probe", "key", "expected bytes", &SeedMetadata::default());
let client = client(&target);
for (status, expected_code) in [(404, "NoSuchKey"), (500, "InternalError"), (403, "AccessDenied")] {
target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
let error = retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
.await
.expect_err("an acknowledged object must not disappear or fail before a later successful read");
assert_eq!(s3_probe_error_code(error.as_ref()), Some(expected_code));
assert_eq!(target.take_requests().len(), 1);
}
for status in [429, 503] {
target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await?;
assert_eq!(target.take_requests().len(), 2);
}
target.inject(Operation::GetObject, FaultAction::ResponseStatus(404), 1);
wait_for_replicated_bytes(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await?;
assert_eq!(target.requests().len(), 2);
Ok(())
}
}
#[tokio::test]
async fn append_single_node_pool_extends_ellipses_volumes() {
let mut env = RustFSTestClusterEnvironment::with_topology(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]]))
@@ -184,7 +184,7 @@ async fn four_node_health_inventory_metrics_and_audit_delivery_are_consistent()
let (status, metrics_body) = admin_request(
&node.url,
Method::GET,
"/rustfs/admin/v3/metrics?n=1&by-host=true&by-disk=true",
"/rustfs/admin/v3/realtime?n=1&by-host=true&by-disk=true",
None,
&dist.cluster.access_key,
&dist.cluster.secret_key,
@@ -257,7 +257,7 @@ async fn node_admin_body(dist: &DistCluster, node: usize, path: &str) -> TestRes
}
async fn http_put_counts(dist: &DistCluster, node: usize) -> TestResult<[u64; 2]> {
let body = node_admin_body(dist, node, "/rustfs/admin/v3/metrics?types=512&by-host=true&n=1").await?;
let body = node_admin_body(dist, node, "/rustfs/admin/v3/realtime?types=512&by-host=true&n=1").await?;
let sample: RealtimeMetrics = serde_json::from_str(body.lines().next().ok_or("empty HTTP metrics stream")?)?;
assert!(sample.errors.is_empty(), "HTTP metrics returned errors: {:?}", sample.errors);
let http = sample.aggregated.http.ok_or("HTTP metrics missing at WARN log level")?;
@@ -400,17 +400,36 @@ async fn verify_write_observations_during_peer_failure(dist: &DistCluster, bucke
let observations = storage["info"]["observations"]
.as_array()
.ok_or("recovery omitted observations")?;
let disks = storage["info"]["disks"].as_array().ok_or("recovery omitted disks")?;
Ok(observations.len() == 4
&& observations
.iter()
.all(|item| item["status"] == "succeeded" && item["cached"] == false))
.all(|item| item["status"] == "succeeded" && item["cached"] == false)
&& disks.len() == 16
&& disks.iter().all(|disk| {
disk["state"].as_str().is_some_and(|state| state.eq_ignore_ascii_case("ok"))
&& disk["runtimeState"]
.as_str()
.is_some_and(|state| state.eq_ignore_ascii_case("online"))
}))
},
"peer probes recover to fresh successful observations",
"peer probes and remote disks recover to fresh successful observations",
)
.await?;
wait_for_ready(&dist.cluster).await?;
assert!(observed_put(dist, 0, bucket, "recovered").await?.is_success());
assert_eq!(http_put_counts(dist, 0).await?, [baseline[0][0] + 1, baseline[0][1] + 2]);
let recovered_before = http_put_counts(dist, 0).await?;
wait_until(
Duration::from_secs(90),
|| async { Ok(observed_put(dist, 0, bucket, "recovered").await?.is_success()) },
"recovered PUT after remote disk and lock convergence",
)
.await?;
let recovered_after = http_put_counts(dist, 0).await?;
assert_eq!(recovered_after[0], recovered_before[0] + 1);
assert!(
recovered_after[1] >= baseline[0][1] + 2,
"recovery retries must retain the two intentional sub-quorum failures: before={recovered_before:?}, after={recovered_after:?}"
);
for node in 0..dist.cluster.nodes.len() {
assert_object_bytes(&dist.client(node)?, bucket, "healthy-0", b"write-observation").await?;
assert_object_bytes(&dist.client(node)?, bucket, "recovered", b"write-observation").await?;
@@ -26,7 +26,7 @@
use super::harness::{
DistCluster, DistLayout, TestResult, assert_object_bytes, cluster_admin_ok, enable_versioning, get_object_bytes, put_object,
unique_bucket, wait_until,
s3_probe_client, s3_probe_error_code, sha256_hex, unique_bucket, wait_until,
};
use crate::common::{
AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, init_logging,
@@ -111,6 +111,7 @@ async fn create_iam_user(dist: &DistCluster, user: &str, secret: &str, policy_na
}
async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, label: &str) -> TestResult {
let client = s3_probe_client(client);
wait_until(
CREDENTIAL_TIMEOUT,
|| {
@@ -119,8 +120,18 @@ async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, l
let key = key.to_string();
let body = body.clone();
async move {
put_object(&client, &bucket, &key, body).await?;
Ok(true)
match put_object(&client, &bucket, &key, body).await {
Ok(()) => Ok(true),
Err(error)
if matches!(
s3_probe_error_code(error.as_ref()),
Some("AccessDenied" | "InvalidAccessKeyId" | "SlowDown" | "ServiceUnavailable")
) =>
{
Ok(false)
}
Err(error) => Err(error),
}
}
},
label,
@@ -129,6 +140,7 @@ async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, l
}
async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8], label: &str) -> TestResult {
let client = s3_probe_client(client);
wait_until(
CREDENTIAL_TIMEOUT,
|| {
@@ -137,8 +149,24 @@ async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8
let key = key.to_string();
let expected = expected.to_vec();
async move {
let got = get_object_bytes(&client, &bucket, &key).await?;
Ok(got == expected)
match get_object_bytes(&client, &bucket, &key).await {
Ok(got) if got == expected => Ok(true),
Ok(got) => Err(format!(
"{label}: object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
sha256_hex(&expected),
sha256_hex(&got)
)
.into()),
Err(error)
if matches!(
s3_probe_error_code(error.as_ref()),
Some("AccessDenied" | "InvalidAccessKeyId" | "SlowDown" | "ServiceUnavailable")
) =>
{
Ok(false)
}
Err(error) => Err(error),
}
}
},
label,
@@ -146,6 +174,57 @@ async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8
.await
}
#[cfg(test)]
mod retry_tests {
use super::*;
use crate::common::build_test_s3_config;
use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction, Operation};
#[tokio::test]
async fn iam_polls_only_retry_auth_propagation_and_availability() -> TestResult {
let target = FakeS3Target::start().await?;
target.create_bucket("upgrade-retry-probe");
let client = Client::from_conf(build_test_s3_config(
target.endpoint(),
FAKE_ACCESS_KEY,
FAKE_SECRET_KEY,
None,
"upgrade-retry-test",
));
for status in [403, 503] {
target.inject(Operation::PutObject, FaultAction::ResponseStatus(status), 1);
wait_for_put(&client, "upgrade-retry-probe", "key", b"body".to_vec(), "IAM PUT").await?;
assert_eq!(target.take_requests().len(), 2);
target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
wait_for_bytes(&client, "upgrade-retry-probe", "key", b"body", "IAM GET").await?;
assert_eq!(target.take_requests().len(), 2);
}
target.inject(Operation::PutObject, FaultAction::ResponseStatus(500), 1);
let error = wait_for_put(&client, "upgrade-retry-probe", "key", b"body".to_vec(), "IAM PUT")
.await
.expect_err("IAM convergence must not hide InternalError on PUT");
assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
assert_eq!(target.take_requests().len(), 1);
target.inject(Operation::GetObject, FaultAction::ResponseStatus(500), 1);
let error = wait_for_bytes(&client, "upgrade-retry-probe", "key", b"body", "IAM GET")
.await
.expect_err("IAM convergence must not hide InternalError on GET");
assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
assert_eq!(target.take_requests().len(), 1);
let error = wait_for_bytes(&client, "upgrade-retry-probe", "key", b"wrong bytes", "IAM GET")
.await
.expect_err("IAM convergence must not hide a payload mismatch");
assert!(error.to_string().contains("bytes mismatch"));
assert_eq!(target.requests().len(), 1);
Ok(())
}
}
async fn seed_history_and_iam(dist: &DistCluster) -> TestResult<UpgradeSeed> {
let history_bucket = unique_bucket("upg-hist");
let versioned_bucket = unique_bucket("upg-ver");

Some files were not shown because too many files have changed in this diff Show More