Compare commits

...

156 Commits

Author SHA1 Message Date
Chris 1e7065101d feat: Capture offline CPU profiles in serving process (#8245)
feat: capture offline CPU profiles in serving process
2026-09-29 22:37:10 +08:00
Chris 380e98a42c fix(test): synchronize top-disk fixture writes with sampling (#8243) 2026-09-29 21:07:45 +08:00
Chris a88225d208 fix(ci): reduce duplicate work and preserve reliable test failures (#8233)
* fix(ci): reduce duplicate work and preserve reliable test failures

* fix(ci): retain protocol evidence and repair stale test fixtures

* test(connect): honor parent deadline during API fixture readiness

* fix(ci): reserve IO capacity for state writer proofs

* test(connect): align RPC fixtures with service capture contracts

* test(connect): cover pinned service capture failures
2026-09-29 21:06:45 +08:00
Chris 1cca0dd25c feat(connect): select offline key for client performance (#8242) 2026-09-29 20:49:59 +08:00
Chris b0a54f0e5c feat(connect): capture offline network performance in server (#8240)
* feat(connect): capture offline network performance in server

* fix(connect): use storage facade in RPC test

* fix(connect): keep RPC test body behind storage facade
2026-09-29 19:41:43 +08:00
Chris 83a2d8b02d feat(connect): capture RPC activity in offline server runtime (#8237) 2026-09-29 18:16:54 +08:00
Chris f661651aad feat(connect): capture API activity in offline server runtime (#8234) 2026-09-29 17:18:26 +08:00
Chris 9298991d14 fix(scanner): restart a finished mixed sweep under its requested plan (#8231)
A bucket sweep that ends mixed clears its position and records the finishing cycle's plan as started. The next cycle requests a new plan whenever the bucket was written in between, so a fresh sweep inherited a stale started plan and ended mixed again. On a continuously written bucket no sweep could ever certify and the census kept the old root.

Start a fresh verification sweep under the requested plan when no durable position remains. Resumed sweeps keep their started plan, so a clean tail still cannot certify an old prefix.

Refs #7108
2026-09-29 16:48:58 +08:00
Chris 3c7d85420c fix(test): control clocks in Connect network fixtures (#8228) 2026-09-29 16:48:34 +08:00
Chris 6a7880c9fc feat(connect): capture offline lock metrics in server (#8230) 2026-09-29 16:29:07 +08:00
hector 0e58bd09d0 fix(ci): run security chain evidence from the lane's own checkout (#8229)
The security workflow checks the repository out into rustfs-repo/ (#7212)
but its chain evidence steps still invoke
scripts/functional_chain_evidence.py relative to the workspace root, which
on the persistent shared runner resolves to a stale checkout left by another
job. The evidence gate then compares that checkout's HEAD against the chain
workflow_sha and rejects the lane before any case runs
("lane checkout differs from chain workflow source").

Run the evidence script from the lane's own checkout so ROOT resolves to
rustfs-repo/, whose HEAD is exactly the chain-pinned workflow_sha.
2026-09-29 16:09:31 +08:00
Hauser fcb60e7322 fix(rpc): preserve walk_dir missing-path errors (#8226)
## Related Issues

rustfs/rustfs#8217 and rustfs/backlog#2683.

## Summary of Changes

Reviewed 539ed275db against d2175d1e1e. No findings. Capable callers requesting missing-path reporting recover explicit pre-output FileNotFound or VolumeNotFound errors; partial-output and other failures continue to fail the stream.

## Verification

Two independent source reviews completed across all lenses below; the frozen diff matches Git and passes diff whitespace checks. Traced the authenticated handler, bounded first-chunk preflight, cancellation ownership, HTTP status/token classification and quorum error handling. The new tests cover ordinary streaming, typed missing errors, byte preservation and mixed missing/I/O failures.

Author-reported focused tests and Docker reproduction were not rerun or independently audited in this review. The reported Docker comparison predates the final report_notfound-only gate; its scope is not distributed acceptance of the final head.

## Impact

Correctness: no findings; missing errors remain typed only before output. Security/trust: no findings; authentication, body digest and operation/status/token restrictions remain. Compatibility: no findings; legacy clean EOF remains, with the documented old-server limitation. Concurrency/durability: no findings; receiver ownership cancels an abandoned producer. Simplicity: no findings. Coverage: no blocking gap identified. Performance: no findings; preflight retains one bounded chunk and ordinary report_notfound=false streams start immediately.

## Additional Notes

This review does not establish that the patch fixes any current main CI failure. Full main CI and release acceptance remain separate gates.
2026-09-29 15:29:59 +08:00
Hauser 0a61f4a81e fix(heal): park healthy legacy MRF intents (#8225)
## Related Issues

rustfs/backlog#2682 and rustfs/rustfs#8192.

## Summary of Changes

Reviewed ca970f55ec against d2175d1e1e. No blocking code findings. The hold requires a completed healthy legacy check and an exact incarnation, lease, object, version and scope; it preserves the durable journal and rechecks on restart.

## Verification

Two independent source reviews completed, covering the lenses below. The frozen diff matches Git and passes diff whitespace checks. Reported Cargo and Docker results were not rerun or independently audited in this review.

One factual correction to the PR description: `legacy_sigkill_replay_repairs_without_releasing_unverified_responsibility` uses the default four-member fixture, with three replicas before restoration and four afterward (`mrf_partial_write_test.rs:559,567`). That named regression exercises the production manager path, but is not EC12+4. Other tests in the file use sixteen disks.

## Impact

Correctness: no findings; proofless legacy health never becomes a verified receipt. Security/trust: no findings; exact identity fences remain. Compatibility: no findings; journal encoding remains unchanged. Concurrency/durability: no findings; replay retains one checkpointed owner and replacement generations become retryable. Simplicity: no findings. Coverage: no blocking gap identified, with the test-scope correction above. Performance: no findings; held entries leave the retry index without adding per-admission queue scans.

## Additional Notes

The current main journal failure concerns DecodeFailure, which follows the separate ECDecode task path. This review does not establish that this PR fixes that failure or the scanner-cycle failure, and does not establish a passing main CI or release gate.
2026-09-29 15:28:46 +08:00
Chris d2175d1e1e fix(test): keep bucket disk faults across reconnects (#8224) 2026-09-29 14:07:30 +08:00
Chris 2019715d1a Fix fresh capacity probes for formatted local disks (#8222) 2026-09-29 12:23:06 +08:00
Chris 31f05a44af fix(test): use expect_err for offline state root rejection (#8220) 2026-09-29 11:22:16 +08:00
Chris a1724f3dbe feat(connect): capture signed offline service health (#8219) 2026-09-29 10:50:23 +08:00
Chris c62979a45b fix(connect): secure new offline state roots (#8218) 2026-09-29 10:13:58 +08:00
Chris e14eacc5ed fix(rpc): keep legacy format reads in their original namespace (#8216) 2026-09-29 09:24:39 +08:00
Chris 47af5565c7 fix(ci): preserve cluster startup failure evidence (#8213) 2026-09-29 08:50:09 +08:00
Chris f0ce628a7f feat(connect): capture offline native threads in service process (#8215) 2026-09-29 08:48:56 +08:00
Chris 32cc1eb76c Capture offline memory profile from the running service (#8214)
feat(connect): capture offline memory from the running service
2026-09-29 08:43:25 +08:00
Chris 84fe13989e fix(ecstore): retain read quota and control reserve test hedging (#8211) 2026-09-29 07:16:31 +08:00
Chris c8f42b8dca fix(ci): isolate scanner deadline and expose test failure details (#8210)
* fix(ci): isolate scanner deadline fixture and expose readiness errors

* test(ecstore): report unexpected capacity reservation errors
2026-09-29 07:16:17 +08:00
Chris c9acf01fd0 fix(tier): reread mutation intents after lock contention (#8209) 2026-09-29 07:16:01 +08:00
Chris 87e9a84a4e fix(ecstore): preserve RPC status when cloning storage errors (#8207)
* fix(ecstore): preserve RPC status when cloning storage errors

* test(heal): retain clone coverage without redundant ownership
2026-09-29 07:15:46 +08:00
Chris af0ae798f8 fix(test): serialize replacement_bucket_metadata heal test under parallel load (#8212) 2026-09-29 07:07:31 +08:00
dependabot[bot] a2bdcf497b build(deps): bump the dependencies group with 14 updates (#8176) 2026-09-29 01:59:07 +08:00
Chris 5564932f38 fix(heal): retry cancelled internode RPC operations (#8205) 2026-09-29 01:58:35 +08:00
Chris 8778d55e46 fix(ci): preserve diagnostics for rio storage test failures (#8204) 2026-09-29 00:15:26 +08:00
Chris 5ebc1e480b test: stabilize heal timing and improve heartbeat replay coverage (#8150)
fix: preserve heartbeat replay and stabilize timing tests
2026-09-28 22:07:15 +08:00
Chris 4d782141d8 fix(test): add missing unavailable_drives field to readiness test initializers (#8200)
Commit 151103a609 added the  field to
StorageReadinessDetails but did not update two test initializers in
readiness.rs, causing E0063 compile errors in --all-targets builds.
2026-09-28 21:26:42 +08:00
Hauser 3bc36acf7f fix(notify): persist Docker queue stores and expose open errors (#8201)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-28 21:24:12 +08:00
Chris 4a91c75152 fix(ci): restore checks and multipart migration retries (#8196)
* fix(ci): restore multipart lint and full E2E membership checks

* fix(ecstore): preserve multipart completion identity during migration
2026-09-28 21:22:49 +08:00
Chris 2648776be7 fix: accept catalog files in health service artifact (#8198)
* fix: accept catalog files in health service artifact

* ci: retain configured health acceptance runner
2026-09-28 20:45:26 +08:00
RustFS 9442e89f5f fix(iam): require an explicit permission for force-delete (#8154)
* fix(iam): require an explicit permission for force-delete

A force-delete header no longer inherits s3:* or consoleAdmin. Bucket
force-delete requires s3:ForceDeleteBucket whenever the header is present,
and recursive object force-delete requires s3:ForceDeleteObject. A plain
delete keeps the existing checks.

Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* test(e2e): keep force-delete header names static

The bucket force-delete helper must pass a static header name into the
SDK request mutator.

Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* test(e2e): move the force-delete header into the request mutator

The SDK request customizer requires a static header name owned by the
closure.

Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* fix(iam): keep force-delete out of NotAction grants

NotAction now uses plain wildcard matching, so NotAction "s3:*" still
excludes force-delete. An Allow statement grants s3:ForceDeleteObject or
s3:ForceDeleteBucket only when its Action list names the action; a
NotAction-only Allow never does. The rule applies to both IAM and bucket
policy statements.

Also build the invalid-header errors with S3Error::with_message to keep
the s3s footprint at its baseline, and fix a clippy single_match.

---------

Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-09-28 19:08:33 +08:00
RustFS fc5609bbb0 fix(s3): make retried CompleteMultipartUpload idempotent (#8153)
Record the upload id on the completed object so a lost-response retry
returns that object's ETag instead of NoSuchUpload, while a different
part list or a replaced object keeps the existing errors.

Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-28 19:08:10 +08:00
Chris 64bd224b3b fix(connect): accept the health-era heartbeat and measured Top windows (#8195)
Two Connect tests fail on main in every Test and Lint lane.

Heartbeat: #8155 replaced "today's capabilities minus
profile.memory.service@1" with "the pre-health set minus it". That drops
the release that advertised health.check.service@1 but not yet in-service
memory profiling, so a pending heartbeat persisted by that release now
fails validation and the runtime stops. Accept that release as its own
frozen list.

Top disk/net: execute_*_job sets max_duration_millis to the requested
duration, then the capture compared the measured sleep against it. The
timer only overshoots, so a job that ran exactly as requested was
rejected with LimitExceeded whenever the overshoot reached 1 ms. Report
the authorized window, as Top API already does; validate_capture bounds
it by the limit.
2026-09-28 19:07:39 +08:00
Chris 71f8b607ad fix(ci): restore mainline and scheduled test reliability (#8187)
* fix(ci): restore mainline and scheduled test reliability

* fix(ci): provide GitHub CLI for CPU acceptance

* ci: provide Docker for CPU service acceptance

* ci: provision Python and Docker for OIDC validation

* ci: restore hosted runners for Docker validation

* ci: use verified MinIO release packages for interop

* ci: preserve host ownership of MinIO fixtures

* fix(ci): correct diagnostic limits and isolate startup checks

* test(connect): include object CLI failure details

* test(readiness): initialize unavailable drive diagnostics
2026-09-28 19:07:29 +08:00
GatewayJ cef0c61532 fix(table-catalog): preserve data sequences during file rewrites
Merge the reviewed fix from pull request #8104.
2026-09-28 18:51:58 +08:00
Justin Bradfield fc325bcc56 fix(s3): report x-amz-mp-parts-count on GetObject with partNumber
Merge the reviewed fix from pull request #8115.
2026-09-28 18:51:48 +08:00
cxymds 4a46f5ff3d fix(s3): serve bucket websites on dedicated domains
Merge the reviewed fix from pull request #8183.
2026-09-28 18:48:21 +08:00
cxymds e0a973bf4e fix(s3): preserve prefix listing pagination and marker metadata
Merge the reviewed fix from pull request #8181.
2026-09-28 18:47:03 +08:00
hector 671238458c ci: fail the pool lane when expected step markers are missing
Merge the reviewed fix from pull request #8190.
2026-09-28 18:46:12 +08:00
Chris 151103a609 fix(ecstore): recover remote disks after transient stalls
Merge the approved fix from pull request #8149.
2026-09-28 18:40:30 +08:00
Nikita Bakun cc7d5e3f5f fix(replication): stop sending versionId query once a target rejects it
Merge the approved fix from pull request #8109.
2026-09-28 18:40:20 +08:00
yi111 d556d4dc94 fix(admin): report a missing policy or user as 404 NoSuchResource, not 500 InternalError
Merge the approved fix from pull request #8127.
2026-09-28 18:40:10 +08:00
chapman 6300fbe74f fix: map IAM not-found errors to 404 in InfoCannedPolicy and RemoveUser
Merge the approved fix from pull request #8134.
2026-09-28 18:38:20 +08:00
Chris 6ebc78d115 fix(ci): run profile acceptance on Docker runners (#8191) 2026-09-28 16:14:07 +08:00
Chris 5f4d5e8fe8 fix(ci): run CPU acceptance with Docker
Use the Docker-enabled runner required by the connected CPU service-job test.
2026-09-28 15:38:36 +08:00
Chris e4520f70a1 fix(ci): verify CPU acceptance identity without gh
Use authenticated curl for the pinned release run and artifact checks on sm-standard-2.
2026-09-28 15:13:03 +08:00
hector f54945a4c4 fix(ci): repair pool budgets and Connect test failures (#8155)
* ci: give the pool run a real wall-clock budget

The pool suite budgets up to 24h each for rebalance and decommission
(REBALANCE_TIMEOUT / DECOMMISSION_TIMEOUT in rustfs_pool_expand.sh),
but the workflow step capped it at 45 minutes. Four consecutive runs
died identically: steps 1-7 all PASS, then the rebalance wait was
killed at exactly 45:12 - chain 35680308150, standalone 35702803577,
chain 35757773373, chain 36283120750 - with rebalance at completed=1/4
(~21 minutes in), so a full pass has never been observed.

Make the budget an input (default 240 minutes: covers the observed
rebalance pace plus one decommission pass) and document why. The suite
keeps failing the job through its [POOL-STEP] marker adjudication.

* fix(ci): align pool job budget and timeout contract tests

* fix(connect): preserve legacy heartbeats and isolate I/O tests

---------

Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: RustFS <hello@rustfs.com>
2026-09-28 14:26:21 +08:00
Hauser 4beac12439 fix(connect): return profile capture result directly (#8182)
* fix(connect): return profile capture result directly

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* style(connect): normalize profile capture formatting

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(tests): use async mutex for thread profiles

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(connect): widen diagnostic timing budgets

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-28 14:25:27 +08:00
Chris 6523ab8d9f fix(ci): verify GNU packages with CPU catalogs
Match the archive-member check to the GNU pyroscope catalog packaging condition.
2026-09-28 13:23:46 +08:00
Chris 0609e7ce14 fix(connect): admit complete release CPU symbol catalogs
Bound the release catalog to 32 MiB and 200,000 symbols based on the GNU build measurement; keep complete names and reject catalogs beyond either limit.
2026-09-28 12:09:26 +08:00
Chris afb9ddabb4 fix(helm): route Gateway API traffic to the chart service (#8145)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-28 02:26:56 +00:00
RustFS 2e014d25b3 fix: bound restarted-peer stalls on quorum reads and writes (#8152)
A restarted peer can accept a pooled connection and never send response
headers, so HttpReader::open waited past the client body timeout before
the body-stall timer or erasure hedge could run. Bound that header wait
by the stall timeout and retry the open once on a fresh connection.

After write quorum, MultiWriter still waited out the full disk stall
for a silent peer, which matches the client timeout. Give remaining
writers one second, then drop them so the caller returns.

Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2026-09-28 02:22:41 +00:00
Chris 719000460b ci(connect): match GNU build runner in acceptance checks (#8180) 2026-09-28 09:08:25 +08:00
hector 528a368144 ci: refresh the preview-release contract for the current build workflow (#8178)
The contract check asserts exact lines in build.yml, and five recent
build-workflow PRs drifted it out of sync, so Security Audit's Workflow
Pin Report job fails for every PR and every push to main since Sep 27
17:34Z (first failure on push run 36337450094). Refresh the assertions
to the current, intentional shapes - contract intent is preserved and
in fact tightened:

- #8160 added PROFILE_ARGS to the cross/native cargo build lines.
- #8171 added 'should_publish == true' as a stricter prefix on the R2
  publication guard and the create-release / upload-release-assets /
  publish-release / update-latest-version conditions.

Verified locally: the full check (including the docker and helm
workflow contract tests) passes against main's files with these
assertions.
2026-09-28 08:54:33 +08:00
Chris d791d7b761 ci(connect): use Node 25 for acceptance workflows (#8179) 2026-09-28 08:52:32 +08:00
Chris 1a251de4f3 feat(connect): execute signed disk diagnostics in the service (#8177) 2026-09-28 08:36:49 +08:00
Chris e634df8611 fix: collect offline Top disk from the running service (#8174)
fix(connect): capture offline disk IO in the running service
2026-09-28 01:35:40 +08:00
Chris f5fbb5f4e1 ci: verify Top disk in the serving process (#8173) 2026-09-28 01:34:29 +08:00
Chris 5d2d539423 ci: default manual builds to artifacts without publishing (#8171)
Default manual builds to artifacts without publishing
2026-09-28 01:10:52 +08:00
Chris acebc9649d feat(connect): capture service runtime profiles over local IPC (#8168)
feat(connect): export service runtime profiles over local IPC
2026-09-28 00:49:45 +08:00
Chris e33542b0c2 fix(scanner): recover after cycle-state persistence failures (#8148)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 16:39:41 +00:00
Chris 2ee0ad1704 Collect bounded metrics from the serving Tokio runtime (#8167)
feat(connect): collect bounded service runtime profiles
2026-09-28 00:20:19 +08:00
Chris 4c5fc3c991 Use pinned release catalogs for Connect CPU acceptance (#8166) 2026-09-28 00:18:41 +08:00
Chris 6d2b1c629e Generate CPU symbol catalogs from release executables (#8165) 2026-09-28 00:12:36 +08:00
RustFS de484b93e0 fix: admit cold reads and readiness at read quorum (#8156)
EC 2+2 still meets read quorum with two of four nodes up. A survivor
that had not cached a bucket returned 503 on GET, HEAD, and List, and
/health/ready left the Service once write quorum was lost. Reads and
Service membership now follow read quorum and shared locks. Writes and
/minio/health/cluster still require write quorum and exclusive locks.

Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 13:58:55 +00:00
hector 4f61b007da ci: build linux-aarch64 natively on sm-standard-4-arm and verify every Linux package (#8160)
Route the two linux-aarch64-* release legs to the dedicated arm64 runner
and build them natively (cross: false) instead of cross-compiling via
cargo-zigbuild on x86_64 hosts.

With native legs now executable, extend the packaged-artifact checks that
were previously gated to x86_64-unknown-linux-gnu only:

- Every Linux leg runs rustfs --version and rustfs-cli --help from its own
  package, proving the artifact matches the runner architecture.
- The packaged-console runtime smoke (server boot + console HTTP 200) also
  covers aarch64-unknown-linux-gnu, so both architectures get full runtime
  verification. Musl legs run the binary checks but skip server boot for
  now.
2026-09-27 13:40:23 +00:00
cxymds 1dd354233c test(connect): sync heartbeat capability expectations (#8161) 2026-09-27 12:22:48 +00:00
Chris 6eb14a11d0 fix(ci): run health acceptance on Docker-capable workers (#8158) 2026-09-27 14:26:28 +08:00
Chris 3667eea893 fix(ci): validate health binary without file utility (#8157) 2026-09-27 14:22:49 +08:00
hector e67bbd7bd3 ci: run the chain orchestration jobs on the smoke-testing runner (#8147)
* ci: run the chain orchestration jobs where the gh CLI exists

#8112 moved these jobs to the sm-standard-2 pool, whose images ship
without the gh CLI (a known property of the build fleet, see #7572).
Last night's chain died in prepare before any suite ran:
resolve_functional_candidate.py:34 does subprocess.run(["gh", "api"])
and got FileNotFoundError; complete-chain and the hourly
functional-chain-health canary fail the same way. On Sep 22/23 the same
jobs ran green on GitHub-hosted runners.

Move prepare, complete-chain and the health canary to ubuntu-latest.
The suite lanes stay on smoke-testing.

* ci: run the chain orchestration jobs where the gh CLI exists

#8112 moved these jobs to the sm-standard-2 pool, whose images ship
without the gh CLI (a known property of the build fleet, see #7572).
Last night's chain died in prepare before any suite ran:
resolve_functional_candidate.py:34 does subprocess.run(["gh", "api"])
and got FileNotFoundError; complete-chain and the hourly
functional-chain-health canary fail the same way. On Sep 22/23 the same
jobs ran green on GitHub-hosted runners.

Move prepare, complete-chain and the health canary to ubuntu-latest.
The suite lanes stay on smoke-testing.

* ci: point the chain orchestration jobs at the smoke-testing runner

Per maintainer decision, consolidate them onto the same runner as the
test lanes. Verified on rustfs-smoke-testing as the runner user:
gh 2.45.0, jq 1.7.

* ci: point the chain orchestration jobs at the smoke-testing runner

Per maintainer decision, consolidate them onto the same runner as the
test lanes. Verified on rustfs-smoke-testing as the runner user:
gh 2.45.0, jq 1.7.
2026-09-27 08:26:43 +08:00
Chris 8d033c85d8 fix(ci): install GitHub CLI for Connect health acceptance (#8146) 2026-09-27 08:04:54 +08:00
Chris b7420fe5f2 fix(ci): bootstrap release upload CLIs (#8144)
fix(ci): bootstrap release upload CLIs on self-hosted runners
2026-09-27 08:00:48 +08:00
Chris 5cb1c9e8cb fix(architecture): route health.rs storage imports through storage_api facade (#8143)
The health diagnostics module imported directly from
crate::storage::storage_api, bypassing the architecture migration
guardrail. Add a connect facade module to rustfs/src/storage_api.rs
and redirect the imports. Also fix a clippy::redundant-guards lint
in the coarse-flags match arm.
2026-09-27 07:53:54 +08:00
Hiroaki KAWAI 20356ab709 fix(scanner): build raw enumeration indexes incrementally (#8114)
* fix(scanner): build raw enumeration indexes incrementally

* test(scanner): iterate over restart test entries directly

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 07:49:59 +08:00
Chris 2c5c43b0e0 test(connect): expose typed drive measurement failure context (#8099)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 05:11:47 +08:00
Chris cf096b3c22 fix(connect): pace diagnostic network payload sends (#8096)
* fix(connect): pace diagnostic network payload sends

* test(connect): exercise native network pacing entry

---------

Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 05:11:35 +08:00
Chris 82024907c5 test: add health service acceptance workflow (#8140) 2026-09-27 03:55:17 +08:00
Chris e2785846d0 ci: keep gh-dependent release jobs on GitHub-hosted runners (#8137)
The sm-standard-2 fleet has no gh CLI (see #7572), so moving the release publication jobs onto it in #8112 broke every tag release: the 1.0.1-preview.12 build failed in Create GitHub Release with 'gh: command not found'. Move create-release, upload-release-assets, publish-release, cleanup-preview-releases, and package.yml resolve/package back to ubuntu-latest.
2026-09-27 03:37:23 +08:00
Chris 31ac243865 feat(connect): add bounded health service job 2026-09-27 02:50:49 +08:00
Chris 8741bb77ea fix(storage): weight automatic multipart admission by part size (#8118)
* fix(storage): weight automatic multipart admission by part size

* fix(ci): restore filesystem runner capabilities and typos dependency

* fix(ci): make release guard portable and spell out part variables

* ci: restore sm-standard-2 runners for io_uring and distributed e2e jobs

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-26 22:35:39 +08:00
cxymds 212b20f890 fix(ecstore): admit cold bucket metadata at read quorum (#8120)
* fix(ecstore): admit cold bucket metadata at read quorum

Encode a durable creation-commit state in bucket metadata so committed Object Lock buckets can use read quorum for cold loads while pre-physical creation intents remain fail-closed. Drain commit-write fan-out, keep write quorum for uncommitted intents, and add regression coverage for exact, below, migrated, and partial-create quorum boundaries.

* fix(ecstore): fence bucket creation-commit persistence

Address review findings on the creation-commit proof.

Persist the proof only under the bucket metadata transaction fence: at bucket creation, and through a fenced migration that re-reads the authoritative metadata and revalidates physical presence at write quorum before writing. This stops a stale snapshot from reverting an acknowledged configuration update or outliving a delete/recreate.

Establish commitment when Object Lock is enabled on an existing bucket, inside the same configuration mutation, so a healthy cluster no longer rejects object operations with ErasureWriteQuorum.

* fix(ecstore): keep commit fence error message stable

The error(format!) ratchet requires a stable Display for quorum bucketing; use a fixed message instead of embedding the bucket name.

* test(ecstore): reuse canonical Object Lock fixture in regression tests

The s3s footprint ratchet is shrink-only. Use the existing ENABLED_OBJECT_LOCK_CONFIG static instead of naming s3s DTO types in store tests.

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-26 21:40:40 +08:00
hector 7d3d0b6c2b ci: align the matrix contract with the 36-combination sweep (#8136)
Companion to rustfs/auto-testing#109: the 4x1 topology is removed from
MATRIX_TOPOS because the product rejects a 1-drive-per-node distributed
pool at startup (FATAL BelowMinimum). The unfiltered sweep is now 36
combinations / 300 cases; update the contract totals and comments so an
unfiltered sweep is still held to an exact expected case count.
2026-09-26 20:31:32 +08:00
hector d317bb3274 ci: run the KMS Vault lanes on ubuntu-latest again (#8133)
#8112 replaced ubuntu-latest with sm-standard-2 across workflows. Both nightly
KMS Vault lanes need a working Docker daemon; the sm-standard-2 pool does not
provide one, so both lanes die in seconds on 'Docker is not available' while
the package compiles and publishes fine - and the functional chain never fires
because its gate requires a successful build run. Restore the lanes to
ubuntu-latest, as documented in the lane comment before #8112.
2026-09-26 19:12:18 +08:00
Hauser 5cac685e9a feat(ecstore): integrate io_uring backend improvements (#8105) 2026-09-26 17:14:17 +08:00
Hauser d8bf268885 fix(scanner): bound SNSD deep scans and checkpoint cloning (#8126)
* fix(scanner): bound SNSD deep scans and checkpoint cloning

* ci: run mount-dependent jobs on hosted VMs

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
2026-09-26 06:06:50 +00:00
hector fdd753a1dd ci: run distributed e2e on ubuntu-latest to restore tmpfs mounts (#8128) 2026-09-26 12:00:26 +08:00
Chris de80e8a673 fix(ci): make shared checks portable across runners (#8124)
* fix(ci): isolate monitor argument checks from runner tools

* fix(ci): install the Typos action download dependency

* fix(ci): make release policy matching portable across awk variants
2026-09-26 10:55:06 +08:00
RustFS 4cf45e9ed2 Update bug report storage requirements: allow SAN/JBOD, recommend XFS (#8125)
docs(github): clarify supported storage in bug reports

State that local disks, SAN/Fibre Channel volumes, and JBOD are supported,
recommend XFS, and keep the existing unsupported remote/shared filesystem rule.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-09-26 10:43:33 +08:00
Chris 9f8acf688c feat(github): add issue forms with storage support checks (#8123) 2026-09-26 10:07:49 +08:00
Chris 96fc26cd6a fix(release): defer installation updates until artifacts are live (#8122) 2026-09-26 09:17:15 +08:00
hector edcc81a8fd ci: replace ubuntu-latest runner with sm-standard-2 across workflows (#8112)
* ci: replace ubuntu-latest runner with sm-standard-2 across workflows

* ci: keep scheduled-validation monitors on hosted runners

The freshness and watchdog jobs report stalled scheduled validations.
Running them on the same sm-standard-2 pool means a pool outage stalls
the monitors too, so nothing reports it.

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-09-25 10:02:45 +08:00
Hauser 286bbb8e1f chore: upgrade workspace dependency releases (#8107)
* chore: upgrade workspace dependencies

* chore: refresh dependency lockfile

* fix(rebalance): avoid waiting on active entry gate during stop preparation
2026-09-24 21:24:14 +08:00
cxymds 0913686b12 fix(heal): detect stale delete-marker metadata (#8102) 2026-09-24 11:16:02 +08:00
George Melikov df64242f64 fix(ecstore): resume scan_dir past a dash-suffixed sibling directory (#8071)
A flat (recursive) listing resumed inside "s/" re-emitted everything in
its sibling "s-x/". scan_dir dropped the entries before forward_to by
comparing directory names without their trailing slash, where "s" sorts
before "s-x", while the keys they stand for sort the other way round:
'-' (0x2d) is below '/' (0x2f), so all of "s-x/..." precedes "s/...".
The drain stopped at "s" and kept "s-x".

The next page then started with keys at or before the marker; the
listing layer filtered all of them out, found no more candidates and
answered IsTruncated=false. A bucket of 104,137 objects with backup
directories named "<id>" and "<id>-rollbacks" listed as 5,000; two such
directories of 1,200 keys each listed as 2,000.

Compare every remaining entry as the key prefix it stands for, slash
included, and keep it only when it sorts at or after forward_to or
contains it. The remainder of forward_to is taken before `current` is
trimmed, so this also holds below the bucket root; retain also drops
entries when all of them precede forward_to, which the drain never did.
2026-09-24 07:13:30 +08:00
Chris d3b75e695e test(connect): add scheduler receipt acceptance workflow (#8078)
* test(connect): add scheduler receipt acceptance workflow

* chore(deps): upgrade crates and fix faster-hex advisory (#8098)

* test(connect): isolate scheduler acceptance workflow

---------

Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-24 03:34:27 +08:00
Chris dde1aa77a1 fix(connect): preserve diagnostic schedules across restart (#8092) 2026-09-24 03:28:37 +08:00
Chris be01e513be feat(connect): sample memory within the running service (#8091)
* feat(connect): sample memory within the running service

* test(connect): add official memory service acceptance

* test(connect): consume the final memory job without cloning
2026-09-24 03:25:28 +08:00
cxymds 9ab034df62 fix(heal): preserve retryable writer failures (#8089)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-24 02:41:16 +08:00
Tyler Hillery 624463c5ba fix(s3): default CopyObject content-type to binary/octet-stream on REPLACE (#8073)
fix: default CopyObject content-type to binary/octet-stream on REPLACE
2026-09-23 18:04:05 +00:00
Chris 713920dcd6 docs: update license copyright to RustFS, Inc. (#8097) 2026-09-24 00:56:08 +08:00
GatewayJ f219a0aba8 perf(ecstore): parallelize multipart I/O setup and metadata reads (#8085)
* perf(ecstore): parallelize multipart I/O setup and metadata reads

* perf(ecstore): share multipart paths and increase read concurrency

* fix(ecstore): route multipart benchmark through storage API facade
2026-09-23 22:19:23 +08:00
Hauser 311e4306e6 chore(ci): complete action pin and HAProxy upgrades (#8090) 2026-09-23 19:20:39 +08:00
Hauser 7f0f4941cf chore(ci): refresh checkout pin in validated workflows (#8083)
* chore(ci): update checkout pin in validated workflows

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): pin migrated checkout alerts to requested commit

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(ci): pin additional checkout workflows to verified commit (#8084)

* chore(ci): pin more checkout workflows to requested commit

Update six additional workflows to the verified upstream checkout commit without changing their permissions, inputs, or triggers.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(ci): pin functional and OIDC checkout uses

Extend the verified checkout commit pin to functional-chain and OIDC workflows without changing their behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-23 09:27:54 +00:00
cxymds 655bb470b8 fix(heal): retry faulty storage disk errors (#8086) 2026-09-23 07:48:39 +00:00
cxymds 56a4099f99 fix(heal): recreate missing bucket volumes (#8082)
* fix(heal): recreate missing bucket volumes

* fix(heal): use typed error for empty targets
2026-09-23 14:51:24 +08:00
hector ee6de7d786 ci(package): build gnu and musl DEB/RPM variants with distinct file names (#8079)
* ci(package): build gnu and musl DEB/RPM variants with distinct file names

The Build and Release workflow produces four Linux binaries
(x86_64-gnu, aarch64-gnu, x86_64-musl, aarch64-musl), but packaging
only consumed the two gnu artifacts. Add matrix entries for the two
musl artifacts so every release ships all four DEB/RPM variants.

The libc variant is now part of the package file names, which would
otherwise collide between gnu and musl builds of the same version:

- deb: rustfs_<version>_<libc>_<arch>.deb
- rpm: rustfs-<libc>-<version>-<release>.<arch>.rpm

The dpkg Package and rpm Name stay plain "rustfs", so gnu and musl
remain mutually exclusive upgrades of one package rather than
co-installable packages fighting over /usr/bin/rustfs.

Dependency declarations now follow the linkage: gnu binaries
dynamically link glibc and keep Depends: libc6 (>= 2.31) /
glibc >= 2.31; musl binaries are statically linked and declare no
libc dependency. The libc variant is also visible in the package
description.

scripts/release/package_versions.sh gains a LIBC argument and its
contract tests cover both variants plus the invalid-libc cases.

* ci(package): align deb/rpm file names with the zip artifact naming

Rename the package file names so every release asset of one build
shares the same stem as its binary artifact, differing only by
extension:

- before: rustfs_<deb_version>_<libc>_<deb_arch>.deb
          rustfs-<libc>-<rpm_version>-<rpm_release>.<rpm_arch>.rpm
- after:  rustfs-linux-<arch>-<libc>-v<version>.deb / .rpm

e.g. rustfs-linux-x86_64-gnu-v1.0.0.zip,
     rustfs-linux-x86_64-gnu-v1.0.0.deb,
     rustfs-linux-x86_64-gnu-v1.0.0.rpm.

Non-development builds embed the raw release tag (with 'v'), like the
zips; development builds embed dev-<full sha>. The dpkg/rpm versions
(including the '~' prerelease ordering) are unchanged - they live in
the package metadata, and a side effect is that release asset names no
longer contain '~' (which GitHub normalizes to '.').

package_versions.sh now takes the target arch (x86_64|aarch64) instead
of the deb/rpm arch pair; the deb Architecture (amd64/arm64) in the
control metadata still comes from the workflow matrix. The two test
workflows that assemble deb download URLs from a release tag
(rustfs-table-test, rustfs-upgrade-test) are updated to the new name,
which also removes their '~'-to-'.' asset name workaround.
2026-09-23 03:17:47 +00:00
cxymds 1880b42169 fix(heal): preserve decode repair signals and retry startup quorum (#8067)
* fix(startup): retry transient bucket metadata quorum failures

* fix(ecstore): retain recovered shard damage for read repair

* fix(ecstore): reconnect missing read sources before GET

* test(heal): prove exact marker replay preserves history

* fix(ecstore): fence reconnect retries at the deadline

Reject expired reconnect waiters before dispatch and start cooldown on sweep completion, including timeout and cancellation. Cover exact-deadline admission independently of cooldown and preserve the original strict RPC-count regression.
2026-09-22 14:58:31 +00:00
Chris 3f549e26f3 fix(ci): publish version-tagged preview Docker images (#8068) 2026-09-22 13:06:23 +00:00
Dae-Cheol Noh 3c6c88b2e7 fix(amqp): emit S3 events directly in notification records (#8066)
* fix(amqp): emit S3 events directly in notification records

* test(amqp): avoid queued timestamp equality race
2026-09-22 12:16:40 +00:00
Dae-Cheol Noh 1f04a12abf fix(ftps): bound upload memory with multipart streaming (#8064)
* fix(ftps): bound upload memory with multipart streaming

* fix(ftps): keep multipart upload within driver module
2026-09-22 12:15:20 +00:00
cxymds 122abfaae2 feat(integrity): add inventory, audit, and protected migration (#8065)
* feat(integrity): add inventory, audit, and protected migration

* refactor(admin): use gateway facade for integrity handlers

* fix(integrity): sort fingerprint metadata explicitly
2026-09-22 19:23:41 +08:00
cxymds d0ce2f758b fix(ecstore): bound decommission target gate contention (#8061)
* fix(ecstore): bound decommission target gate contention

Target capacity gate contention during pool decommission escalated a
per-object transient into a durable bucket pause: each contended object
failed its bucket entry, which re-ran the whole bucket listing and amplified
attempts on the same objects.

- centralize the decommission capacity failure classification so gate
  contention, benign contention and fatal failures are decided once
- retry target-gate contention inline (bounded, jittered) before a mutation
  is admitted, covering put, part, complete, new-multipart and abort
- defer contended entries to the end of the round instead of failing the
  bucket entry, and require the deferred set to drain before a set completes
- treat a missing object or version, an overwrite and a superseded upload id
  as benign contention that is neither counted as a failure nor escalated
- use full-jitter exponential backoff, capped, for decommission retries
- expose the capacity pause reason, the waiting reason and a cumulative
  pause count in the admin pool status, plus gate-retry and per-object
  attempt metrics

Related: rustfs/backlog#2644

* fix(ecstore): correct deferred replay and metadata compatibility
2026-09-22 19:20:05 +08:00
hector 9c30cc8851 feat(ci): on-demand fault-tolerance matrix sweep workflow (#8058)
Adds rustfs-fault-tolerance-matrix.yml, a manually dispatched workflow that
runs the --matrix topology x EC outage sweep from rustfs/auto-testing (38
combinations, 320 cases). The scenario suite hardcodes --all and a 60-minute
budget, so the exhaustive sweep had no CI entry point.
2026-09-22 13:22:18 +08:00
Hauser 0ff06757c8 fix(obs): support custom OTLP CA bundles (#8060)
Add RUSTFS_OBS_TLS_CA_FILE for additive private CA trust in OTLP HTTP exporters.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-22 13:21:44 +08:00
cxymds e12100b4a1 fix(s3): emit x-amz-expiration as HTTP-date (#8057)
The expiry-date field of the x-amz-expiration response header was
rendered as RFC 3339 (e.g. 2026-10-02T00:00:00Z). AWS SDKs parse this
value with an RFC 822 parser; the Java SDK v1 rejects the ISO-8601
shape, drops the whole header, and logs a WARN from
ObjectExpirationHeaderHandler. S3 specifies HTTP-date (RFC 1123 with a
literal GMT zone), which is also what MinIO emits.

Render via the shared HTTP-date formatter (extracted alongside
format_expires_header) and pin the exact output in the unit test.
2026-09-22 11:18:59 +08:00
hector d7c42d4099 fix(ci): accept the testing-sha staleness fallback in chain evidence (#8056)
The 09-22 nightly chain failed all 12 lanes in seconds at the 'Bind
functional candidate' step:

  ValueError: private script pin differs from chain

resolve_functional_candidate.py's >24h staleness fallback (added by
#8026, made functional by #8041's token fix) legitimately sets
manifest.testing_sha to auto-testing main HEAD, but current_chain()
still required it to equal .config/functional-script-revision.txt -
a check written for the pre-fallback world where the two could never
diverge. Once the fallback finally fired, prepare produced testing_sha
21edcf4 while the pin file still holds 27e9584 and every lane aborted
before checking out the test scripts.

Drop the pin-file comparison and keep what the lane actually needs to
guarantee: testing_sha is a valid commit sha (current_chain), the lane
checked out exactly that sha (record: private_head == testing_sha, kept
as-is), and the health checker validates the same format instead of
re-reading the pin file. Tests updated: a fallback testing_sha that
differs from the pin is accepted; a non-sha testing_sha is rejected.

Verified: python3 -m unittest test_functional_chain
test_functional_chain_health -> 39 tests OK.
2026-09-22 10:25:36 +08:00
hector 4b0118520e test(ci): align FT workflow contract tests with scenario E (44 cases) (#8054)
#8052 renamed the fault-tolerance step to 'Run fault-tolerance
scenarios (A, B, C, C2, D, E)' and raised the evidence gate to 44
cases, but scripts/test_security_workflow.py still expected the old
step name and a 38-case fixture, breaking Quick Checks with three
KeyErrors and one assertion failure. Update DIRECT_TESTS and the
write_result fixture to the new contract.
2026-09-22 00:18:40 +08:00
cxymds 1afaf4491e fix(ecstore): bound decommission capacity gate waits (#8044)
* fix(ecstore): bound decommission capacity gate waits

* fix(ecstore): keep rebalance meta save lock errors retryable

The rebalance metadata retry policy only retries when the source chain still
carries the typed lock error, but resolve_rebalance_meta_save_result collapsed
every failure into a plain string. A transient rebalance.bin write-lock timeout
therefore bypassed retry_rebalance_metadata_access entirely and surfaced as a
hard failure of the periodic and terminal rebalance metadata saves.

Wrap the failure with the data-movement stage context instead, which keeps the
original lock error reachable through rebalance_error_source so the existing
transient lock policy applies. The rendered message is unchanged.

Add a regression test that feeds the wrapped meta save lock timeout through the
retry helper and asserts the retry engages.

* chore(ecstore): refresh error format ratchet baseline

Removing the string-wrapped meta save failure drops one `::other(format!`
call site in the rebalance worker, so the shrink-only baseline must be
regenerated in the same change.
2026-09-21 20:56:26 +08:00
hector bdea44f332 fix(ci): align FT evidence contract with scenario E (44 cases) (#8052)
The fault-tolerance suite now registers and runs scenario E
(rustfs/auto-testing#100): --all covers A, B, C, C2, D, E with 38 + 6
= 44 expected cases. The chain-evidence assertion still hardcoded
[A, B, C, C2, D] and 38 everywhere, which would fail every green FT
run's evidence validation once E is registered.

Bump the scenario list to include E and all count assertions from 38
to 44; step name updated to match. No other lanes reference 38.
2026-09-21 20:13:56 +08:00
cxymds 8ba3309624 fix(rebalance): retry fleet proof before start activation (#8049)
Retry the transient missing fleet capability proof in both the admin start flow and node RPC activation. Return a retryable 503 with Retry-After after rollback when readiness does not converge.
2026-09-21 11:27:58 +00:00
cxymds cd2fa50e50 fix: rename admin metrics endpoint to realtime (#8046) 2026-09-21 19:23:58 +08:00
cxymds 077b961232 fix(obs): default span events to none and gate span list by level (#8047)
JSON log layers synthesized span lifecycle events for every span and
embedded the full ancestor span list on every record. With INFO spans on
object hot paths that produced multi-megabyte lines, which journald
truncated at 48 KiB: the payload was silently dropped and the volume
itself became the dominant load.

Add RUSTFS_OBS_LOG_SPAN_EVENTS (none|close|full) with a default of none,
and keep with_span_list enabled only for verbose (debug/trace) levels so
the ancestor chain stops being re-serialized per record. The request_id
promotion path reads the span scope, not the span list, so it is
unaffected.

Refs rustfs/backlog#2642
2026-09-21 19:15:39 +08:00
cxymds e1bc731331 fix(lifecycle): reject empty expiration actions (#8045)
* fix(lifecycle): reject empty expiration actions

* style: apply rustfmt to lifecycle validation
2026-09-21 15:43:07 +08:00
cxymds bb43ef861a fix(decommission): use owned data for source capacity (#8038) 2026-09-21 15:29:30 +08:00
Chris 4047ed6c8d fix(heal): restore bucket metadata before replacement completion (#8042) 2026-09-21 12:18:06 +08:00
hector ae6bbaef27 fix(ci): give the chain staleness probe a token that can read auto-testing (#8041)
resolve_functional_candidate.py probes rustfs/auto-testing (private) to
age the pinned functional-script revision and fall back to main HEAD
after 24h. The prepare step passed github.token, which cannot see the
private repo, so every nightly chain logged

  staleness probe failed (...exit status 1.); keeping pinned revision

and replayed the 09-14 harness. On the 09-20 nightly that harness died
on the dpkg conffile prompt in all 12 lanes (see rustfs/auto-testing#97)
because the --force-confold and other fixes never reached the chain.

Use PF_TESTING_GH_TOKEN - already required by the other steps in this
workflow - so the probe can actually run and the >24h fallback works.
2026-09-21 11:08:41 +08:00
cxymds a7c875b48f fix(heal): preserve remote corruption and reject incomplete deep scans (#8032)
* fix(heal): reject truncated encoded shards

* test(heal): cover truncated shards across integrity modes (#8035)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): preserve remote corruption and reject incomplete deep scans

* test(admin): tolerate config lock timeout while polling

* fix(test): use admin storage error facade

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-09-21 02:33:03 +00:00
マルコメ 42dd76d3b2 ci(docker): sync the Docker Hub overview from README.md on release (#8034)
Docker Hub's overview is a separate `full_description` field that
`docker push` never touches, so it had drifted into an 8 KB snapshot of
an old README.md that still linked to
https://docs.rustfs.com/introduction.html (now 404).

Add a `sync-dockerhub-description` job to docker.yml that runs after the
images are pushed and republishes README.md from the same commit via
peter-evans/dockerhub-description (pinned to v5.0.0). It reuses the
existing DOCKERHUB_USERNAME / DOCKERHUB_TOKEN credentials, so no new
secrets are needed. Relative links (docs/, CONTRIBUTING.md) are
rewritten to github.com URLs so they resolve on Docker Hub.

Docker Hub caps the field at 25,000 bytes and the action truncates to
fit with only a warning; README.md is at 22,879 bytes today. Read the
published overview back after the sync and fail the job if it hit the
cap, so a truncated overview cannot be published silently.

Fixes #7995

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-21 07:04:47 +08:00
Chris 9aeecad4a9 docs(security): add encrypted ETag advisory lesson (#8040) 2026-09-20 23:52:47 +08:00
Hauser 3738b32df1 chore(deps): update flake.lock (#8033)
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/aff8a0b' (2026-09-10)
  → 'github:NixOS/nixpkgs/a32edd7' (2026-09-17)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/228ecef' (2026-09-12)
  → 'github:oxalica/rust-overlay/26a71e6' (2026-09-19)
2026-09-20 14:54:45 +08:00
Hauser d997b07505 fix(get): skip futile resume for single-disk objects (#8031)
* chore: pin s3s to upstream git revision

Use the upstream s3s git repository at 50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6 for both s3s and s3s-sigv4.

Allow the official s3s git source in cargo-deny so source checks continue to pass.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(get): skip futile resume for single-disk objects

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-20 13:25:08 +08:00
Chris 0632a64f91 fix(ecstore): allow server-side copy at normal read quorum (#8029) 2026-09-20 12:36:53 +08:00
hector cbf6b02905 ci(chain): fall back to auto-testing main HEAD when the pinned harness goes stale (#8026)
The functional chain pins the test harness to the revision recorded in
.config/functional-script-revision.txt, which is only refreshed by
release syncs - it went stale on Sep 16 and the nightly chain has been
replaying a Sep-17-era harness ever since. Last night that meant:
the performance lane died on a dpkg conffile prompt (fix merged as
#88 but never reached the chain), the FT lane died on 'unknown option:
--results-file' (harness/workflow version mismatch), and the
STS-105 signing fix (#94) sat untested.

When the pinned revision's commit is older than 24 hours, fall back to
auto-testing main HEAD so the chain always runs the current harness.
Any staleness-probe failure keeps the pinned revision (fail-safe).
The revision file stays in place as the audit trail and the release
sync continues to manage it.
2026-09-20 11:27:56 +08:00
hector ed2c3eaf77 fix(package): preserve service state across upgrades (#8024)
* fix(package): preserve service state across upgrades

* fix(package): match legacy DEB versions in tilde form

Published prerelease packages carry ~ in the dpkg control version
(package_versions.sh maps the SemVer prerelease - to ~), so the
legacy fallback list written with dots never matched 1.0.0~rc.x and
upgrades away from those DEBs still left the service stopped (#8011).

Fix the legacy glob to the tilde form and update the contract test,
which had enshrined the dot form. Verified on Ubuntu 24.04 systemd
containers: DEB upgrade 1.0.0~rc.5 -> 1.0.1 now keeps the service
running; 1.0.0 -> 1.0.1 and 1.0.1 -> 1.0.2 marker path still pass.

* docs(package): expand /etc/default/rustfs example template

Document the commonly used RUSTFS_* settings as commented examples in
the packaged conffile and point to docs.rustfs.com.
2026-09-20 11:27:41 +08:00
cxymds 22243e791d test(heal): cover stale current delete marker rejoin (#8021) 2026-09-19 19:33:24 +08:00
Hauser c35de00e8a fix(scanner): protect single-disk foreground latency (#8022)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-19 19:32:02 +08:00
cxymds 0e03bcda49 fix: serialize notify runtime module switch reconciliation (#8020) 2026-09-19 13:41:39 +08:00
cxymds c3269ad4c2 fix(heal): close tail-truncation and stale current-marker gaps (#8019)
* fix(heal): verify complete encoded shard tails

* fix(heal): prove stale current marker deletion
2026-09-19 13:41:30 +08:00
Chris 0fcd34c331 fix(ecstore): retry fleet capability probe while the notification system boots (#8017)
Startup finalizes IAM before init_notification_runtime publishes the notification system, and IAM finalization is what starts the fleet capability probe. The first probe pass therefore always failed closed and then slept the full ten-second probe interval. On a single node that left every durable capability, including the durable hard quota fence, unavailable for about ten seconds after /health already reported ok, so SetBucketQuota answered 503 durable quota capability is not confirmed across the cluster during that window.

Publishing the notification system now wakes the probe immediately through a Notify permit, with a 100ms bootstrap poll as the fallback for a wakeup that races the availability check. The probe keeps failing closed while the system is absent and logs the wait once. The four probe futures no longer carry an unreachable notification-system-unavailable arm, and the seven proof slots are revoked through one helper.

Fixes #8014
2026-09-19 13:41:16 +08:00
Chris ddc5f97a04 fix(release): sort pre-release packages below their final release (#8016)
Pre-release RPM versions replaced the SemVer `-` with `_`, which rpm treats as an ordinary segment separator, so `1.0.0_rc.5` compared as newer than `1.0.0` and a host with an rc RPM installed could not `dnf upgrade` to the GA package. DEB previews had the mirror-image problem: only the first `-` became `~`, so `1.0.0~rc.5-preview.2` carried `preview.2` as its Debian revision and sorted above `1.0.0~rc.5`.

Both formats now spell every pre-release separator as `~`, which dpkg and rpm (>= 4.10) both treat as "sorts before anything". Verified with rpm 4.16 (AlmaLinux 9) and dpkg 1.21: `1.0.0~rc.5 < 1.0.0`, `1.0.0~rc.5~preview.2 < 1.0.0~rc.5 < 1.0.0~rc.6`, alpha < beta < rc, and `rc.9 < rc.10`; fpm 1.18 passes `~` through into both package headers, and `dnf upgrade` from a `~rc.5` RPM to the GA RPM succeeds where the `_rc.5` one refused. The test now pins the full ordering contract under both package managers and fails against the previous spelling.

The release-checksum step already normalizes `~` to `.` for every uploaded asset, so RPM assets need no further handling. Hosts that already installed an `_rc`/`_beta` RPM need a one-time `dnf install rustfs-1.0.0` or `rpm -Uvh --oldpackage` to reach GA.

Fixes #8012
2026-09-19 03:22:44 +08:00
cxymds ff62c810b4 fix(tier): gate opaque version recovery cleanup (#8006)
* fix(tier): gate opaque version recovery cleanup

* test(tier): authorize opaque recovery fixtures

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
2026-09-18 21:54:03 +08:00
cxymds 950db7412c fix(heal): inspect stale members during deep scans (#8009)
* fix(heal): inspect stale members during deep scans

* test(heal): prove convergence outcomes
2026-09-18 21:53:31 +08:00
Chris 4e7a8fabb8 fix(ecstore): repair buckets left with only an incarnation sidecar (#8008)
The legacy bucket metadata migration writes the `.bucket-incarnation`
sidecar before `.metadata.bin`. A crash or a lost namespace lease between
the two writes left the bucket with a sidecar and no metadata, and every
later load failed closed with "bucket incarnation sidecar exists without
bucket metadata", so the bucket answered 500 to every request on every
node with no repair path (rustfs/rustfs#8003).

Load the sidecar-only state as a legacy bucket so the migration runs
again. The migration and the force-create path re-read the sidecar under
the transaction lock and keep its incarnation when persisting the
metadata, so the retry never replaces an identity other nodes fenced on.
A retired incarnation is never adopted: it is residue of a deleted
bucket, and re-publishing it would let heal reclaim the new objects.
2026-09-18 18:14:00 +08:00
cxymds b0f4e66c92 fix(tier): model R2 as unversioned storage (#8004) 2026-09-18 10:12:51 +00:00
cxymds e9da1fa074 feat: expose subscriptions used by event destinations (#8001)
* feat: expose event subscriptions by destination

* style: format event subscription handler

* fix: preserve s3 error footprint checks
2026-09-18 16:11:19 +08:00
Hauser 023adb1696 fix(heal): discharge absent durable partial writes (#8000)
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-18 16:07:18 +08:00
Hauser 0f69c073de fix(ecstore): preserve bounded listing completion reason (#7994)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-18 15:41:44 +08:00
cxymds 7a5e1efa95 fix(heal): fail admin heal with unhealthy drives (#7996) 2026-09-18 12:58:24 +08:00
Chris aa7f55946c docs(operations): add cluster and erasure-coding lifecycle runbook (#7998)
Consolidate the operator entry point for planning, storage classes and
EC:0 consequences, expansion, rebalance, decommission, heal, drive
replacement, restart recovery, and the rc CLI mapping. Every claim was
checked against the current handlers, storage-class validation, quorum
arithmetic, and the rustfs/cli admin reference; the runbook links the
normative contracts instead of restating them.

Refs rustfs/backlog#2607, rustfs/backlog#2608
2026-09-18 12:50:14 +08:00
Hauser 785b44a696 fix(replication): harden delete marker HEAD convergence (#7993)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-18 11:05:47 +08:00
Hauser 415bd2310c build(deps): bump dial9 to 0.5.1 (#7991)
Update dial9 and the resolved companion crates to 0.5.1.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-18 09:53:30 +08:00
393 changed files with 35164 additions and 3958 deletions
+22 -13
View File
@@ -1,6 +1,6 @@
---
name: rustfs-release-publish
description: "Run the end-to-end RustFS console gate, version bump, preview validation, human confirmation, and final-tag publication pipeline. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
description: "Run the RustFS console gate, source version bump, preview validation, human confirmation, final-tag publication, and post-release installation and website updates. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
---
# RustFS Release Publish (preview-validated pipeline)
@@ -18,16 +18,18 @@ Pipeline shape:
```
check console main against its latest Release
-> if ahead: publish console -> wait for Release asset + latest API
-> bump RustFS version files to <target> (final version, ONE commit) -> merge
-> bump Cargo.toml and Cargo.lock to <target> -> merge
-> tag <preview-tag> at that commit -> CI green
-> verify preview Release assets -> run binary locally + console checks
-> validate with latest rc client
-> report preview acceptance results -> STOP for explicit human confirmation
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
-> CI deletes the <target>-preview.N Releases (tags kept)
-> verify published images -> publish Helm chart from the validated source
-> update installation references on main -> update rustfs.com announcement
```
On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
On preview validation failure: fix lands on main via normal PR (Cargo versions are already at `<target>`, no new source bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2. Installation references remain on the previous published deliverable throughout preview validation.
## Required inputs
@@ -56,19 +58,21 @@ Rules:
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
## Hard rules
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
- Before preview, bump only Cargo.toml (workspace package and internal dependency versions) and the corresponding workspace members in Cargo.lock, directly to `<target>`. Keep README installation examples, flake.nix, Chart.yaml, and rustfs.spec on the previous published deliverable until Phase 7. Never write a `-preview.N` suffix into these files; preview identifiers belong to validation tags and artifacts.
- Publishing a tag alone does not make an installation target available. Verify the exact image manifest, Release assets, and source archive before advancing references that consume them. Helm packaging derives its versions from the final tag and waits for that image; it does not require an early Chart.yaml bump on main.
- Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
- Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final. Phase 7 updates installation references on main after publication; neither tag moves to that follow-up commit.
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
- Phases run in order; a failure blocks dependent steps. For a preview/source acceptance failure before the final tag exists, land the fix, verify Cargo still matches the confirmed target, and restart from Phase 2 with the next preview at the fixed commit. If main has advanced to another target, resolve that source/version mismatch before another preview.
- Once the final tag exists, preserve its commit. Retry failed publication jobs for that exact tag; do not recreate the tag or restart preview on newer main. If a source change is required after final-tag publication, report the failure and obtain a new target version. Phase 7 failures resume only the failed follow-up after rechecking artifact availability and whether a newer release has superseded it.
- Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
- Confirmation is scoped to the reported `<target>`, `<preview-tag>`, and `PREVIEW_HASH`. A failed or repeated acceptance cycle, including any new preview iteration, invalidates prior confirmation and requires a new one.
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
@@ -84,10 +88,10 @@ Rules:
Read and complete [the Console gate](references/console-gate.md) before Phase 1. Verify the latest published Console asset and exact commit; if Console main is ahead, complete its release and asset verification first. A successful build alone does not satisfy this gate.
## Phase 1 — Version bump to the final target (once)
## Phase 1 — Source version bump to the final target (once)
- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
- If Cargo.toml and all workspace members in Cargo.lock already read `<target>` (e.g. this is a restart after a failed preview), verify both and skip the source bump. Installation versions intentionally differ during preview; do not treat them as leftovers. If a previous preparation advanced installation references to an unavailable target, restore those references to the verified published baseline before selecting the next preview commit.
- Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- Get the PR merged into main. Record the resulting main commit:
```bash
@@ -106,7 +110,7 @@ git push origin "<preview-tag>"
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
@@ -116,7 +120,7 @@ On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
## Phases 4–5 — Local artifact, Console, and rc acceptance
@@ -130,7 +134,7 @@ Continue to Phase 6 only after a new user reply explicitly confirms the reported
## Phase 6 — Publish the final tag on the validated commit
No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
No second source version bump. The final tag goes on the exact commit the preview validated:
```bash
git fetch origin --tags
@@ -145,6 +149,10 @@ git push origin "<target>"
- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
## Phase 7 — Installation references and website announcement
Only after Phase 6 succeeds, complete [post-release updates](references/post-release-updates.md): align installation references on main, then update the existing top banner in `rustfs/rustfs.com`. A failed or incomplete publication leaves both on the previous available version. These follow-up commits never change `PREVIEW_HASH` or either release tag. Track their PR, merge, and deployment states separately from artifact publication; an open PR is not a live website update.
## Output contract
Always report:
@@ -153,4 +161,5 @@ Always report:
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
- Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
- Post-release installation PR and verification results; website banner target, text, link, PR, and observed deployment state. Report any remaining merge authorization or failed deployment explicitly rather than claiming the banner is live.
- Any deviation from this pipeline and why the user approved it.
@@ -0,0 +1,32 @@
# Post-release Installation and Website Updates
Read only after the final non-preview tag has passed Phase 6. Tag creation or a green binary build alone is insufficient: the release assets, source archive, container images, Helm package, and applicable latest channels must be available. Preview tags never enter this phase.
## Installation references
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
## rustfs.com top banner
Use the existing announcement configuration from [website PR #112](https://github.com/rustfs/rustfs.com/pull/112). Do not add a second banner or redesign the header.
1. In an isolated checkout of `rustfs/rustfs.com`, read its current `AGENTS.md`, PR template if present, `data/announcement.ts`, and the consuming header component. Search for an existing announcement PR for this target before creating another.
2. Update `homeAnnouncement` in `data/announcement.ts` after the release and installation availability checks pass. Keep `enabled: true`, `badge: 'New'`, and the existing component. For a stable release, use:
```ts
export const homeAnnouncement = {
enabled: true,
badge: 'New',
message: 'RustFS <target> is now available! Please',
linkText: 'upgrade and try',
href: '/download',
} as const
```
Replace `<target>` with the exact published version. Reserve GA wording for the actual first stable release. For a final alpha/beta/rc tag, explicitly call it a prerelease; use its GitHub Release URL if `/download` does not expose that target. Never announce a `-preview.N` build here.
3. Use the site's current package manager and required checks. For this config-only change, type-check and lint the changed file, build the site, and inspect the generated homepage for the version, visible link text, and destination. Confirm `/download` actually offers the announced version before using that link.
4. Commit, push, and create or update the website PR when included in the authorized release scope. Include the published RustFS Release URL and checks run. Keep merge and production-deployment authority separate; reuse permission already given, but this reference grants none by itself.
5. After an authorized merge/deployment, verify the live `https://rustfs.com/` banner and its destination. If only the PR is ready, report that state and URL; do not report the banner as deployed. A website failure does not invalidate or move the published release tag: retry this phase after resolving the failure.
@@ -1,16 +1,15 @@
---
name: rustfs-release-version-bump
description: "Prepare the version-file and release-asset bump for an exact RustFS alpha/beta/stable target, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
description: "Prepare Cargo versions for an exact RustFS target, or align installation references after its artifacts are published, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
---
# RustFS Release Version Bump
Use this skill to prepare and verify release version files. Commit, push, and PR steps apply only when included in the user's delivery scope; publishing release tags belongs to `rustfs-release-publish`.
Validated baseline: release pattern used in PR `#2957`.
## Required inputs
- Exact target version, for example `1.0.0-beta.4`.
- Stage: `prepare` (default) or `post-release`. Infer `post-release` only when the user or parent release workflow explicitly requests installation updates after publication; a normal version bump means `prepare`.
- Delivery scope: local (`edit/verify`), git (`commit/push`), or GitHub
(`commit/push/PR`). Derive it from the conversation; when unspecified, prepare
and verify locally without blocking on a delivery question.
@@ -25,25 +24,32 @@ Reject any target version containing `-preview`: preview identifiers are tag-onl
- `.github/pull_request_template.md` only when preparing a PR.
- Current branch status and diff against `origin/main`.
## Default release file scope
## Stage boundaries
Treat the following file list as the default checklist for each release bump:
`prepare` updates only:
- `Cargo.toml`
- `Cargo.lock`
`post-release` updates installation and packaging references only:
- `README.md`
- `README_ZH.md`
- `flake.nix`
- `helm/rustfs/Chart.yaml`
- `rustfs.spec`
Only drop a file when the current repository release process clearly no longer requires it.
During preview, installation references intentionally retain the previous published deliverable. Do not sweep them into a source bump to make every version string match. `flake.nix` builds local source, but its package label is aligned with the other packaging references after publication.
Before `post-release` edits, verify that the exact non-preview target has a published GitHub Release, downloadable assets and source archive, and a pullable `rustfs/rustfs:<target>` image manifest. If any prerequisite is missing, report `BLOCKED` and leave installation references unchanged. If a newer deliverable has already superseded this target, do not downgrade installation defaults during a retry.
Helm CI derives chart versions from the triggering tag, so the final tag can retain the previous Chart.yaml values. Update the repository copy only after the target image and published chart are available. Neither post-release changes nor their merge commit may replace the preview-validated final tag.
## Hard release policy
- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
- Helm chart version mapping follows `beta.N -> 0.N.0`.
- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
- Derive Helm chart and app versions with `scripts/helm_chart_version.sh <target>`; the existing mapping is `beta.N -> 0.N.0`, with other target versions retained.
- `rustfs.spec` `Release` uses the prerelease suffix (for example `beta.4`), or `1` for a stable release.
- Do not change these rules without explicit confirmation.
## Step-by-step workflow
@@ -52,22 +58,18 @@ Only drop a file when the current repository release process clearly no longer r
- Use the exact target and delivery scope already supplied; ask only for a missing or ambiguous target or a material release-policy choice.
- Inspect current branch and ensure only release-related files are touched for this task.
2. Update workspace versions
2. Update workspace versions (`prepare` only)
- Bump `[workspace.package].version` in `Cargo.toml`.
- Bump internal workspace crate dependency versions in `Cargo.toml`.
- Update `Cargo.lock` so workspace package versions match target version.
- Re-scan for partial leftovers.
- Re-scan Cargo.toml and workspace members in Cargo.lock for partial leftovers; leave external dependency versions unchanged.
3. Update release assets
3. Update installation references (`post-release` only)
- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
- `flake.nix`: update package version to target version.
- `helm/rustfs/Chart.yaml`:
- `appVersion` = target version.
- `version` follows chart mapping rule, for example:
- `1.0.0-beta.3` -> `0.3.0`
- `1.0.0-beta.4` -> `0.4.0`
- `helm/rustfs/Chart.yaml`: use the app and chart versions returned by `scripts/helm_chart_version.sh <target>`.
- `rustfs.spec`:
- Set `Release` to prerelease suffix (example `beta.4`).
- Set `Version` to the numeric version and `Release` to the prerelease suffix (example `beta.4`), or `1` for a stable release. Verify that `Source0` and the unpacked source directory resolve to the exact published target, including its prerelease suffix when present.
- Add/update top changelog entry with exact format:
- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
- `- Update RPM package to RustFS 1.0.0-beta.4`
@@ -78,15 +80,14 @@ Only drop a file when the current repository release process clearly no longer r
- Changelog version text must match target release version exactly.
4. Verify before shipping
- Run:
- `make pre-commit`
- If `make pre-commit` fails, fix task-attributable failures and rerun affected checks. Report unresolved required checks as `BLOCKED`; do not silently widen scope to fix unrelated issues.
- Follow the root verification tiers for the final diff instead of running a full-workspace gate for version strings.
- For `prepare`, validate Cargo metadata with the updated lockfile and confirm workspace package/internal dependency versions agree. Do not accept a lockfile containing unrelated dependency updates.
- For `post-release`, render the Helm chart and confirm its default image is the verified target; run `scripts/test_helm_chart_version.sh` when chart versions change. Check the README image tags, Nix package version, and expanded RPM source URL against that same target.
- Run `git diff --check` for either stage. Report unresolved required checks as `BLOCKED`; do not silently widen scope to fix unrelated issues.
5. Commit strategy (only when committing is authorized)
- Preferred split when both parts changed:
- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
- `chore(release): align release assets for <version>` for docs and packaging files.
- If user asks for one commit, use one commit.
- Use `chore(release): prepare <version>` for `prepare` and `chore(release): align installation references for <version>` for `post-release`.
- These stages happen on opposite sides of publication; do not combine them in a preview-preparation commit or PR.
- Stage only intended release files; do not include unrelated working tree changes.
6. Push and PR (only for the authorized delivery scope)
@@ -104,13 +105,12 @@ Only drop a file when the current repository release process clearly no longer r
- `git diff --name-only origin/main...HEAD`
- `git diff --stat origin/main...HEAD`
- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
- `make pre-commit`
## Output contract
When using this skill, always report:
- Target version.
- Target version and stage.
- Files changed.
- Any assumptions or uncertainties requiring confirmation.
- Verification result (`PASSED` or `BLOCKED`) with key evidence.
@@ -101,6 +101,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
### SSE and on-disk storage invariants
- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
- `GHSA-wqmc-vjgv-jrpw`: SSE-C persisted and listed the plaintext MD5 as the ETag, allowing list-only users to confirm or brute-force low-entropy content and correlate equal plaintext across keys. Lesson: encrypted-object ETags must not expose deterministic plaintext fingerprints; protect them with per-object key material and keep listing, conditional, copy, and multipart semantics consistent.
### Object Lock and retention invariants
@@ -126,7 +127,7 @@ rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonical
rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
rg -n "HashReader|EncryptReader|try_resolve_etag|ETag|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
```
@@ -145,5 +146,5 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-linux=21ab66fd172018682ce825a53f26aa8f286d57f3f687d8616d1850e56474dcd8
sha256-darwin=21ab66fd172018682ce825a53f26aa8f286d57f3f687d8616d1850e56474dcd8
sha256-linux=6fc377fa1f9f06e065f077c7bd53d04efbccf4453f95dde67afaa47778185377
sha256-darwin=6fc377fa1f9f06e065f077c7bd53d04efbccf4453f95dde67afaa47778185377
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=9e9687ded961aa4e619a15def6e1ee0a5153be05423d009ee7a58126624c83e7
sha256-linux=364666693c2bf258997643c9d87e07e7b7501e58175f190d201d53c81946eb20
sha256-darwin=3b6ff112ff16acbaa4148c4736095a1b6b71cc11af0dd3513ff28c6bdc334ca8
sha256-linux=4e8bfc01870565a0a608a1634e6919fc593ce120a970aa04d974f7edf2675c21
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=83a7dcaffd5a789517ae9f02a224f66a9713937885cff96fca2ad7e216f197ae
sha256-linux=626c10f8c964507ff987b6c86069e9019dc6d2ae7fb02db9be5df5aa8cc5145b
sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1
+1 -1
View File
@@ -1 +1 @@
sha256=548a58d74f3cb3b5a1ab9dcd4d3f8625b9ddebe847a2e14de1d08c3b2e6292eb
sha256=5de6852b2e606eca4ee0d86dbb3b725a3eb8923143fd00df96c549acc36040bd
+4
View File
@@ -24,11 +24,13 @@ TEST_THREADS ?= 1
.PHONY: script-tests
script-tests: ## Run shell script tests
@echo "Running script tests..."
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
./scripts/test_build_rustfs_options.sh
./scripts/test_docker_runtime_timezone.sh
./scripts/test_entrypoint_credentials.sh
./scripts/test_internode_grpc_ab_bench.sh
./scripts/test_object_batch_bench_enhanced.sh
./scripts/test_package_service_scripts.sh
./scripts/test_hotpath_warp_ab_gate.sh
./scripts/test_hotpath_warp_abba.sh
./scripts/test_scanner_validation_harness.sh
@@ -50,6 +52,7 @@ script-tests: ## Run shell script tests
./scripts/check_embedded_secrets.sh --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_scheduled_validation_freshness.py --self-test
@@ -59,6 +62,7 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
$(RUSTFS_PYTHON_BIN) ./scripts/check_object_data_cache_follower_samples.py --self-test
./scripts/validate_object_data_cache_cold_stampede.sh --self-test
+45 -4
View File
@@ -107,6 +107,16 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
test-group = 'ecstore-serial-flaky'
# Serialize the replacement bucket metadata heal test. It builds a real-disk
# hermetic erasure set, writes bucket metadata, then asserts byte-identical
# xl.meta readback after a rejected delayed repair. Under parallel load,
# concurrent ecstore tests mutate the process-wide disk registry and alter
# the serialized timestamp/disk-identity fields, flipping the assertion.
# Preventive serialization only, no retries.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
test-group = 'ecstore-serial-flaky'
# Serialize the metadata-cache generation-retirement pair. Both carry
# #[serial(metadata_cache_invalidation_probe)] — a no-op across nextest's
# process boundary — and assert get_object_metadata_cache generation
@@ -132,10 +142,24 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
test-group = 'embedded-test-ports'
# The real object probe includes bucket creation and cleanup in its short
# measurement budget. Keep competing storage fixtures outside that budget.
# Inventory delivery and real drive/object probes include durable filesystem IO
# in short deadlines. Reserve capacity so unrelated storage fixtures cannot
# exhaust those budgets. Keep the deadlines and assertions unchanged.
[[profile.default.overrides]]
filter = 'package(rustfs) & binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)'
filter = 'package(rustfs) & (binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
threads-required = "num-test-threads"
# These concurrent state writers keep the production 5s lock-acquisition limit
# while their peer completes durable IO. Reserve capacity from unrelated test
# processes while preserving each test's internal two-writer race.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::test_(peer_edit_generations_are_unique_across_nodes|recreated_state_object_allocates_over_the_previous_lifetimes_mark|state_object_lock_serializes_writers_from_separate_nodes|retry_event_persist_must_not_wipe_concurrent_locked_rmw)$/)'
threads-required = "num-test-threads"
# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
# Their existing deadlines, assertions, and retry policy remain in force.
[[profile.default.overrides]]
filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
threads-required = "num-test-threads"
# Serialize the durable manual-transition checkpoint test across nextest's
@@ -320,6 +344,11 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
test-group = 'ecstore-serial-flaky'
# Serialize the replacement bucket metadata heal test under the ci profile too.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
test-group = 'ecstore-serial-flaky'
# Serialize the metadata-cache generation-retirement pair under the ci
# profile too (see the matching default-profile override near the top). Not a
# quarantine: no retries.
@@ -334,7 +363,19 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
test-group = 'embedded-test-ports'
[[profile.ci.overrides]]
filter = 'package(rustfs) & binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)'
filter = 'package(rustfs) & (binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)))'
threads-required = "num-test-threads"
# Keep the same state-writer capacity reservation in CI without changing the
# production lock deadline, internal concurrency, assertions, or retry policy.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::test_(peer_edit_generations_are_unique_across_nodes|recreated_state_object_allocates_over_the_previous_lifetimes_mark|state_object_lock_serializes_writers_from_separate_nodes|retry_event_persist_must_not_wipe_concurrent_locked_rmw)$/)'
threads-required = "num-test-threads"
# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
# Their existing deadlines, assertions, and retry policy remain in force.
[[profile.ci.overrides]]
filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
threads-required = "num-test-threads"
# Serialize the durable manual-transition checkpoint test under the ci profile
-38
View File
@@ -1,38 +0,0 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: ''
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
**Smartphone (please complete the following information):**
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
**Additional context**
Add any other context about the problem here.
+110
View File
@@ -0,0 +1,110 @@
name: Bug report
description: Create a report to help us improve
body:
- type: markdown
attributes:
value: |
### Storage support requirements
**Supported:** local disks, SAN (Fibre Channel) attached volumes, or JBOD-mounted disks. We recommend formatting them with the **XFS** file system.
**Not supported:** NFS, SMB/Samba/CIFS, and other remote or shared filesystems as RustFS data directories. This also applies to the underlying host storage for Docker volumes and Kubernetes PersistentVolumes.
Reports that only reproduce on unsupported storage will be closed without investigation. Please reproduce the issue on supported storage before submitting a bug report.
See [Disk Planning and File System Selection](https://docs.rustfs.com/en/installation/linux/prerequisites-and-service) for details.
- type: checkboxes
id: storage-support
attributes:
label: Supported storage confirmation
options:
- label: I have reproduced this issue with all RustFS data directories on supported storage (local disks, SAN/Fibre Channel, or JBOD; XFS recommended), without NFS, SMB/Samba/CIFS, or other remote or shared filesystem mounts, including storage backing Docker volumes and Kubernetes PersistentVolumes.
required: true
- type: textarea
id: bug-description
attributes:
label: Describe the bug
description: A clear and concise description of what happened.
validations:
required: true
- type: textarea
id: reproduction-steps
attributes:
label: Steps to reproduce
description: Include the commands, configuration, or actions needed to reproduce the problem.
placeholder: |
1. Start RustFS with ...
2. Run ...
3. Observe ...
validations:
required: true
- type: textarea
id: expected-behavior
attributes:
label: Expected behavior
description: A clear and concise description of what you expected to happen.
validations:
required: true
- type: input
id: rustfs-version
attributes:
label: RustFS version
description: Provide the release version, container image tag, or commit hash.
validations:
required: true
- type: dropdown
id: deployment-method
attributes:
label: Deployment method
description: How are you running RustFS? Add any relevant details in Additional context.
options:
- Binary
- Docker / Docker Compose
- Kubernetes / Helm
- Built from source
- Other
- type: textarea
id: storage-mounts
attributes:
label: Data directory filesystems and mounts
description: |
Identify the filesystem type and mount source for every RustFS data directory.
On Linux, include output from `findmnt -T /path/to/data -o TARGET,SOURCE,FSTYPE,OPTIONS` for each data path; on other systems, provide equivalent mount information.
For Docker, identify the backing host path and filesystem, not just the container path or volume name.
For Kubernetes, identify the PV, StorageClass/CSI driver, and actual backing filesystem; a PVC name alone is not sufficient.
Remove credentials and other sensitive information before submitting.
placeholder: |
RustFS data paths:
Filesystem types and mount sources:
Mount command output:
Docker host paths or Kubernetes volume backing storage (if applicable):
validations:
required: true
- type: input
id: operating-system
attributes:
label: Server operating system and architecture
description: Include the operating system version and CPU architecture where RustFS is running.
placeholder: "For example: Ubuntu 24.04, x86_64"
validations:
required: true
- type: textarea
id: client-environment
attributes:
label: Client environment
description: Include the S3 client or SDK and version. For console issues, include the browser, operating system, and desktop or mobile device details.
- type: textarea
id: logs
attributes:
label: Relevant logs
description: Paste relevant log output. Remove credentials and other sensitive information before submitting.
render: shell
- type: textarea
id: screenshots
attributes:
label: Screenshots
description: If applicable, drag and drop screenshots here to help explain the problem.
- type: textarea
id: additional-context
attributes:
label: Additional context
description: Add any other context, such as deployment topology or relevant configuration. Remove credentials and other sensitive information before submitting.
+1
View File
@@ -0,0 +1 @@
blank_issues_enabled: false
-20
View File
@@ -1,20 +0,0 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
@@ -0,0 +1,38 @@
name: Feature request
description: Suggest an idea for this project
body:
- type: checkboxes
id: request-scope
attributes:
label: Request scope
description: |
RustFS data directories backed by NFS, SMB/Samba/CIFS, or other remote or shared filesystems are unsupported, including those behind Docker volumes or Kubernetes PersistentVolumes.
Troubleshooting requests for these deployments will be closed without investigation. Use this form for new or improved functionality.
options:
- label: This is a feature request and does not ask for troubleshooting an unsupported remote or shared filesystem deployment.
required: true
- type: textarea
id: problem-description
attributes:
label: Problem or use case
description: Describe the problem or use case that motivates this feature request.
placeholder: "For example: I am trying to ... but ..."
validations:
required: true
- type: textarea
id: proposed-solution
attributes:
label: Proposed solution
description: A clear and concise description of what you would like to happen.
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
description: Describe any alternative solutions or features you have considered.
- type: textarea
id: additional-context
attributes:
label: Additional context
description: Add any other context or drag and drop screenshots about the feature request here.
+2 -7
View File
@@ -19,7 +19,7 @@ runs:
using: composite
steps:
- name: Install quality tools
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
with:
tool: |
ripgrep@15.2.0
@@ -100,12 +100,7 @@ runs:
- name: Check test wiring
shell: bash
run: |
python3 ./scripts/check_test_wiring.py --self-test
python3 ./scripts/check_scheduled_validation_freshness.py --self-test
python3 ./scripts/test_security_workflow.py
python3 ./scripts/test_nightly_candidate.py
python3 ./scripts/check_test_wiring.py
run: python3 ./scripts/check_test_wiring.py
- name: Check no planning docs committed
shell: bash
+3 -1
View File
@@ -107,9 +107,11 @@ runs:
- name: Install cargo-nextest
if: inputs.install-test-tools == 'true'
uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
with:
tool: cargo-nextest@0.9.138
- name: Setup Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
with:
# false is rust-cache's own default. With true, cleanup.ts returns
# *before* pruning ~/.cargo/registry/src, and config.ts archives the
@@ -1,61 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Architecture Migration Rules
on:
pull_request:
types: [ opened, synchronize, reopened, closed ]
branches: [ main ]
paths:
- "ARCHITECTURE.md"
- "docs/architecture/**"
- "scripts/check_architecture_migration_rules.sh"
- ".github/workflows/architecture-migration-rules.yml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
architecture-migration-rules:
name: Architecture Migration Rules
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install ripgrep
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
with:
tool: ripgrep@15.2.0
- name: Check architecture migration rules
run: ./scripts/check_architecture_migration_rules.sh
+19 -10
View File
@@ -24,8 +24,11 @@ on:
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
- 'scripts/security/check_performance_ab_workflow.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_tier_artifact_workflow.sh'
@@ -40,8 +43,11 @@ on:
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
- 'scripts/security/check_performance_ab_workflow.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_tier_artifact_workflow.sh'
@@ -71,7 +77,7 @@ jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Explain cancellation run
@@ -83,11 +89,11 @@ jobs:
cargo-deny:
name: Cargo Deny
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 20
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -106,7 +112,7 @@ jobs:
# default is now "false", but state it explicitly — see
# scripts/security/check_cache_save_if.sh.
- name: Setup Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
with:
# Same reasoning as the setup composite: true archives every
# dependency's unpacked source tree.
@@ -116,7 +122,7 @@ jobs:
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install cargo-deny
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
with:
tool: cargo-deny
@@ -126,11 +132,11 @@ jobs:
workflow-pin-report:
name: Workflow Pin Report
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 5
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -158,9 +164,12 @@ jobs:
- name: Check package version contract
run: ./scripts/test_package_versions.sh
- name: Check package service lifecycle contract
run: ./scripts/test_package_service_scripts.sh
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
if: github.event_name == 'pull_request' && github.event.action != 'closed'
permissions:
@@ -168,7 +177,7 @@ jobs:
pull-requests: write
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -201,7 +210,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+120 -87
View File
@@ -27,7 +27,8 @@
# schedule and manual dispatch build the full platform matrix.
#
# Manual Parameters:
# - build_docker: Build and push Docker images (default: true)
# - publish: Publish binaries/releases for manual runs (default: false)
# - build_docker: Advisory only; manual runs do not trigger Docker publishing
# - platforms: Comma-separated platform IDs or 'all' (default: all)
name: Build and Release
@@ -55,12 +56,16 @@ on:
- cron: "13 1 * * 0" # Weekly on Sunday 01:13 UTC
workflow_dispatch:
inputs:
publish:
description: "Publish to download channels and GitHub Releases (false keeps only workflow artifacts)"
required: false
default: false
type: boolean
build_docker:
# Advisory only. docker.yml triggers on workflow_run and its job-level
# condition requires the triggering event to be a tag push, so a manual
# dispatch of this workflow never produces images regardless of this
# value. Kept because the summary step reports it; wiring it up would
# mean teaching docker.yml's version parser a second event shape.
# value. Publishing remains controlled by the explicit publish input.
description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
required: false
default: true
@@ -88,17 +93,18 @@ jobs:
# Build strategy check - determine build type based on trigger
build-check:
name: Build Strategy Check
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_publish: ${{ github.event_name != 'workflow_dispatch' || inputs.publish == true }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -172,7 +178,7 @@ jobs:
prepare-platform-matrix:
name: Prepare Platform Matrix
needs: build-check
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
outputs:
matrix: ${{ steps.select.outputs.matrix }}
@@ -208,9 +214,9 @@ jobs:
all='{"include":[
{"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
{"target_id":"linux-aarch64-musl","os":"sm-standard-4","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":"","features":""},
{"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
{"target_id":"linux-aarch64-gnu","os":"sm-standard-4","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope"},
{"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""}
]}'
@@ -257,7 +263,7 @@ jobs:
matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0
@@ -402,6 +408,8 @@ jobs:
- name: Build RustFS
shell: bash
env:
BUILD_TARGET: ${{ matrix.target }}
run: |
# Force rebuild by touching build.rs
touch rustfs/build.rs
@@ -418,14 +426,37 @@ jobs:
# Build one binary at a time so release LTO links cannot exhaust a
# self-hosted runner by running concurrently.
for binary in "${BINARY_NAMES[@]}"; do
PROFILE_ARGS=()
if [[ "$binary" == "rustfs" && "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
# The in-process profiler resolves names from the final executable.
# Retain function symbols without enabling DWARF source information.
PROFILE_ARGS=(--config 'profile.release.strip="none"')
fi
if [[ "${{ matrix.cross }}" == "true" ]]; then
# All cross targets in the matrix are Linux; zigbuild handles them.
cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" -p rustfs --bin "$binary"
cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
else
cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" -p rustfs --bin "$binary"
cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin "$binary"
fi
done
- name: Generate the CPU symbol catalog
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
shell: bash
run: |
cargo metadata --locked --format-version 1 --filter-platform '${{ matrix.target }}' \
--features rustfs/pyroscope > "$RUNNER_TEMP/cpu-catalog-metadata.json"
features=$(jq -ce '
(.packages[] | select(.name == "rustfs") | .id) as $id |
.resolve.nodes[] | select(.id == $id) | .features | map(select(. != "default")) | sort
' "$RUNNER_TEMP/cpu-catalog-metadata.json")
release_dir="target/${{ matrix.target }}/release"
# These task-owned outputs may remain in a restored target cache.
rm -f "$release_dir/rustfs.cpu-symbol-catalog.json" "$release_dir/rustfs.cpu-symbol-catalog.sha256"
cargo run --locked -p rustfs-connect-symbol-catalog -- \
"$release_dir/rustfs" "$(git rev-parse HEAD)" '${{ matrix.target }}' "$features" \
"$release_dir/rustfs.cpu-symbol-catalog.json" > "$release_dir/rustfs.cpu-symbol-catalog.sha256"
- name: Create release package
id: package
shell: bash
@@ -516,6 +547,10 @@ jobs:
BINARY_FILES+=("rustfs-cli")
fi
if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256)
fi
# Verify the binaries exist before packaging
for binary in "${BINARY_FILES[@]}"; do
if [[ ! -f "$binary" ]]; then
@@ -625,7 +660,7 @@ jobs:
echo "🔧 Build type: ${BUILD_TYPE}"
echo "📊 Version: ${VERSION}"
- name: Verify packaged Linux CLI
- name: Verify packaged Linux binaries
if: matrix.platform == 'linux'
shell: bash
run: |
@@ -634,21 +669,27 @@ jobs:
package_file="${{ steps.package.outputs.package_file }}"
archive_members="$(unzip -Z1 "$package_file" | sort)"
expected_members=$'rustfs\nrustfs-cli'
target="${{ matrix.target }}"
features="${{ matrix.features }}"
if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then
expected_members+=$'\nrustfs.cpu-symbol-catalog.json\nrustfs.cpu-symbol-catalog.sha256'
fi
if [[ "$archive_members" != "$expected_members" ]]; then
echo "Linux package must contain exactly rustfs and rustfs-cli" >&2
echo "Linux package does not contain the expected files" >&2
printf '%s\n' "$archive_members" >&2
exit 1
fi
if [[ "${{ matrix.target }}" == "x86_64-unknown-linux-gnu" ]]; then
cli_dir="$(mktemp -d)"
trap 'rm -rf "$cli_dir"' EXIT
unzip -q "$package_file" rustfs-cli -d "$cli_dir"
"$cli_dir/rustfs-cli" --help >/dev/null
fi
# Every Linux target builds natively, so executing the binaries here
# proves each artifact matches its runner architecture.
cli_dir="$(mktemp -d)"
trap 'rm -rf "$cli_dir"' EXIT
unzip -q "$package_file" rustfs rustfs-cli -d "$cli_dir"
"$cli_dir/rustfs" --version
"$cli_dir/rustfs-cli" --help >/dev/null
- name: Verify packaged console
if: matrix.target == 'x86_64-unknown-linux-gnu'
if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
shell: bash
run: |
set -euo pipefail
@@ -702,14 +743,14 @@ jobs:
exit 1
- name: Upload to GitHub artifacts
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ steps.package.outputs.package_name }}
path: "rustfs-*.zip"
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}
- name: Upload to Cloudflare R2
if: env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
if: needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
@@ -726,12 +767,33 @@ jobs:
fi
# The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted
# images did). Install it on demand so R2 uploads survive a fresh runner
# instead of hard-failing here.
# images did). Install the official CLI bundle on demand so R2 uploads
# survive a fresh runner without relying on a distro package.
if ! command -v aws >/dev/null 2>&1; then
echo "aws CLI not found on runner; installing..."
if command -v apt-get >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
aws_arch="$(uname -m)"
case "$aws_arch" in
x86_64) aws_arch="x86_64" ;;
aarch64|arm64) aws_arch="aarch64" ;;
*)
echo "❌ Unsupported Linux architecture for AWS CLI: $aws_arch"
exit 1
;;
esac
aws_install_root="${RUNNER_TEMP:-$PWD}/aws-cli"
rm -rf "$aws_install_root"
mkdir -p "$aws_install_root"
curl --fail --location --retry 3 \
"https://awscli.amazonaws.com/awscli-exe-linux-${aws_arch}.zip" \
--output "$aws_install_root/awscliv2.zip"
unzip -q "$aws_install_root/awscliv2.zip" -d "$aws_install_root"
"$aws_install_root/aws/install" \
--install-dir "$aws_install_root/aws-cli" \
--bin-dir "$aws_install_root/bin"
export PATH="$aws_install_root/bin:$PATH"
aws --version
elif command -v brew >/dev/null 2>&1; then
brew install awscli
else
@@ -770,70 +832,41 @@ jobs:
name: Build Summary
needs: [ build-check, build-rustfs ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Build completion summary
shell: bash
env:
# dispatch input via env: free-form text must not be pasted into the
# script for bash to evaluate.
INPUT_BUILD_DOCKER: ${{ github.event.inputs.build_docker }}
SHOULD_PUBLISH: ${{ needs.build-check.outputs.should_publish }}
BUILD_TYPE: ${{ needs.build-check.outputs.build_type }}
VERSION: ${{ needs.build-check.outputs.version }}
BUILD_STATUS: ${{ needs.build-rustfs.result }}
EVENT_NAME: ${{ github.event_name }}
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
echo "🎉 Build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo ""
# Check build status
BUILD_STATUS="${{ needs.build-rustfs.result }}"
echo "📊 Build Results:"
echo " 📦 All platforms: $BUILD_STATUS"
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview artifacts are published in a GitHub prerelease"
echo "⏭️ Preview releases do not update latest channels"
;;
"development")
echo "🛠️ Development build artifacts have been uploaded to OSS dev directory"
echo "⚠️ This is a development build - not suitable for production use"
;;
"release")
echo "🚀 Release build artifacts have been uploaded to OSS release directory"
echo "✅ This build is ready for production use"
echo "🏷️ GitHub Release will be created in this workflow"
;;
"prerelease")
echo "🧪 Prerelease build artifacts have been uploaded to OSS release directory"
echo "⚠️ This is a prerelease build - use with caution"
echo "🏷️ GitHub Release will be created in this workflow"
;;
esac
echo ""
echo "🐳 Docker Images:"
if [[ "$BUILD_TYPE" == "preview" ]]; then
echo "⏭️ Preview tags do not publish Docker images"
elif [[ "$INPUT_BUILD_DOCKER" == "false" ]]; then
echo "⏭️ Docker image build was skipped (binary only build)"
echo "Build type: $BUILD_TYPE"
echo "Version: $VERSION"
echo "Build result: $BUILD_STATUS"
if [[ "$SHOULD_PUBLISH" != "true" ]]; then
echo "Artifact-only run: no download channels or GitHub Releases are updated."
echo "Successful platform packages are available as workflow artifacts."
elif [[ "$BUILD_STATUS" == "success" ]]; then
echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
echo "Publishing enabled; check the upload and release jobs for their results."
else
echo "❌ Docker image build will be skipped due to build failure"
echo "Build incomplete; check individual platform jobs for any completed uploads."
fi
if [[ "$EVENT_NAME" == "push" && "$GITHUB_REF" == refs/tags/* && "$BUILD_STATUS" == "success" ]]; then
echo "Docker publishing is eligible through the downstream workflow."
else
echo "This run does not trigger Docker publishing."
fi
# Create GitHub Release for every valid release tag, including previews
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
@@ -842,7 +875,7 @@ jobs:
release_url: ${{ steps.create.outputs.release_url }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0
@@ -893,20 +926,20 @@ jobs:
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
@@ -987,8 +1020,8 @@ jobs:
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: sm-standard-2
timeout-minutes: 30
steps:
- name: Update latest.json
@@ -1047,8 +1080,8 @@ jobs:
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
@@ -1085,8 +1118,8 @@ jobs:
cleanup-preview-releases:
name: Cleanup Preview Releases
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
@@ -1140,7 +1173,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+10 -10
View File
@@ -15,16 +15,16 @@
# Sole writer of the Rust dependency caches that ci.yml restores.
#
# Why this is a separate workflow rather than steps inside ci.yml: ci.yml's
# concurrency group cancels in-progress runs on main pushes, and merges land far
# faster than its 70-minute pipeline. Measured over 15 consecutive main pushes:
# concurrency group originally cancelled in-progress main runs, while merges
# landed faster than its 70-minute pipeline. Over 15 consecutive main pushes:
# 12 cancelled, 2 failed, 0 succeeded. A cancelled run never reaches
# Swatinem/rust-cache's post step (cache-on-failure does not cover cancellation),
# so the writer lanes were saving nothing and every PR paid a cold restore —
# 11.8-20.9 minutes of "Setup Rust environment" against 0.7-3.4 warm.
#
# Splitting cache writing out of the test pipeline lets ci.yml keep cancelling
# superseded runs (which is correct — nobody needs test results for a commit
# that is already three merges behind) while the caches still get written.
# Keep cache writing separate from validation: only these jobs publish the
# complete feature closure for each key. Main validation now also finishes its
# running baseline, while superseded PR attempts can still be cancelled.
#
# The group below deliberately does NOT cancel in progress; see the comment on
# it for how that bounds concurrency and why it is scoped by event.
@@ -109,7 +109,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -141,7 +141,7 @@ jobs:
- name: Upload cargo timings report
if: inputs.emit_timings
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: cargo-timings-ci-dev
path: target/cargo-timings/
@@ -199,7 +199,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -230,7 +230,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -260,7 +260,7 @@ jobs:
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
+110 -79
View File
@@ -47,12 +47,12 @@ permissions:
contents: read
# Concurrency groups are scoped per event so different triggers never cancel
# each other: PR pushes cancel the previous run of that PR, main pushes keep
# latest-wins semantics among themselves, and scheduled runs always complete
# (a shared group used to let every merge kill the weekly scheduled run).
# each other. PR pushes cancel superseded attempts; main and scheduled runs
# finish so a busy merge stream cannot starve the complete baseline. For main,
# GitHub retains one running run and replaces the pending run with the latest.
concurrency:
group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
CARGO_TERM_COLOR: always
@@ -63,7 +63,7 @@ jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Explain cancellation run
@@ -72,12 +72,12 @@ jobs:
classify-changes:
name: Select CI scope
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
outputs:
mode: ${{ steps.scope.outputs.mode }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 2
persist-credentials: false
@@ -98,24 +98,28 @@ jobs:
typos:
name: Typos
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install typos download dependency
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends wget
- name: Typos check with custom config file
uses: crate-ci/typos@37bb98842b0d8c4ffebdb75301a13db0267cef89 # master
uses: crate-ci/typos@512fc24f32f44ab01972217aaaf3dc86ec234d53 # v1.50.2
# Fail early with compile-free checks for every pull request.
quick-checks:
name: Quick Checks
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -124,7 +128,7 @@ jobs:
test-and-lint:
name: Workspace Test and Lint
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
@@ -132,7 +136,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# Checkout otherwise writes the token into .git/config, where a PR's
# own build.rs or proc-macro could read it back out.
@@ -142,11 +146,9 @@ jobs:
uses: ./.github/actions/setup
with:
rust-version: stable
# Every lane in this workflow reads its cache and none writes it.
# cache-warm.yml is the sole writer for all four keys: this workflow
# cancels superseded runs on main, and a cancelled run never reaches
# rust-cache's post step, so writing from here saved nothing (12 of 15
# consecutive main-push runs were cancelled). See rustfs/backlog#1600.
# Every lane reads its cache; cache-warm.yml remains the sole writer
# with the matching feature closure. Tests never compete to replace
# shared caches, including when a PR attempt is cancelled.
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
@@ -261,7 +263,7 @@ jobs:
- name: Upload test reports and diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: junit-test-and-lint-${{ github.run_number }}
path: |
@@ -277,21 +279,10 @@ jobs:
- name: Check log-analyzer rule anchors
run: ./scripts/check_log_analyzer_rules.sh
# Explicit gate for migration-critical suites. These tests already ran in
# the full nextest pass above; a single filtered nextest invocation keeps
# the named gate without rebuilding or re-running them one package at a time.
#
# The gate selects tests by name substring (data_movement / rebalance /
# decommission / source_cleanup / delete_marker), so renames can silently
# thin it. The script owns the filter expression and first verifies the
# selected-test count against the committed floor in
# .config/migration-gate-floor.txt before running the gate; renames or
# removals must update that file consciously (see the script header).
# Kept on the default profile (no --profile ci): a second --profile ci run
# would clobber target/nextest/ci/junit.xml, and none of these tests are
# quarantined so they gain nothing from the ci profile's retry overrides.
- name: Run rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh
# Preserve the migration floor and require successful execution evidence
# from the workspace run, without compiling or executing its subset again.
- name: Verify rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
# This gate builds into a fresh target directory to isolate the E2E root.
# Give its cold build a separate budget from workspace tests and migration proofs.
@@ -305,7 +296,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -344,7 +335,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -391,7 +382,7 @@ jobs:
- name: Upload ILM test diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
path: |
@@ -408,7 +399,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -427,6 +418,7 @@ jobs:
run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
- name: Run rio-v2 feature tests
shell: bash
env:
# Match the main nextest lane's #5394 link-I/O guard. A cold feature
# cache otherwise fans out enough rust-lld processes to exhaust this
@@ -436,9 +428,25 @@ jobs:
# --profile ci so the quarantine list (and its junit flaky markers)
# covers this leg too; the default profile is the local no-retry
# profile and silently ignored quarantined flakes here (rustfs#6703).
cargo nextest run --profile ci -p rustfs -p rustfs-ecstore --features rio-v2
mkdir -p artifacts/test-and-lint
rm -f target/nextest/ci/junit.xml
./scripts/ci/resource_sampler.sh start nextest-rio-v2
trap './scripts/ci/resource_sampler.sh stop' EXIT
cargo nextest run --profile ci -p rustfs -p rustfs-ecstore --features rio-v2 \
2>&1 | tee artifacts/test-and-lint/nextest.log
cargo test -p rustfs --doc --features rio-v2
- name: Upload rio-v2 test reports and diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: junit-test-and-lint-rio-v2-${{ github.run_number }}
path: |
target/nextest/ci/junit.xml
artifacts/test-and-lint
retention-days: 3
if-no-files-found: error
connect-short-credential-boundary:
name: Connect Short Credential Boundary
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
@@ -449,7 +457,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -504,13 +512,9 @@ jobs:
runs-on: sm-standard-4
timeout-minutes: 90
strategy:
# On a PR, one failing protocol leg is enough to know the PR is not ready,
# so stop the sibling leg instead of paying another ~40 minutes for it.
# Everywhere else (main pushes, the merge queue, the weekly schedule) keep
# the full signal: there we want to know whether swift AND sftp are broken,
# not just whichever failed first. This is the only part of the early-stop
# work that also covers fork PRs, since it needs no token.
fail-fast: ${{ github.event_name == 'pull_request' }}
# Preserve both independent results when one protocol fails, so diagnosis
# and a focused fix do not require rebuilding an interrupted sibling.
fail-fast: false
matrix:
features:
- name: swift
@@ -521,7 +525,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -541,6 +545,8 @@ jobs:
cargo clippy -p rustfs -p rustfs-protocols --all-targets ${{ matrix.features.flags }} -- -D warnings
- name: Run tests with ${{ matrix.features.name }}
id: protocol-tests
shell: bash
env:
# Keep feature-test linking under the same bounded concurrency as the
# main nextest lane; Clippy is metadata-only and needs no such limit.
@@ -549,11 +555,26 @@ jobs:
# --profile ci so the quarantine list (and its junit flaky markers)
# covers this leg too; the default profile is the local no-retry
# profile and silently ignored quarantined flakes here (rustfs#6703).
cargo nextest run --profile ci -p rustfs -p rustfs-protocols ${{ matrix.features.flags }}
mkdir -p artifacts/protocol-tests
rm -f target/nextest/ci/junit.xml
cargo nextest run --profile ci -p rustfs -p rustfs-protocols ${{ matrix.features.flags }} \
2>&1 | tee artifacts/protocol-tests/nextest.log
- name: Upload protocol test reports and diagnostics
if: >-
always() && contains(fromJSON('["success", "failure", "cancelled"]'), steps.protocol-tests.outcome)
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: junit-test-and-lint-${{ matrix.features.name }}-${{ github.run_number }}-${{ github.run_attempt }}
path: |
target/nextest/ci/junit.xml
artifacts/protocol-tests
retention-days: 3
if-no-files-found: warn
build-rustfs-debug-binary:
name: Build RustFS Debug Binary
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 30
@@ -561,7 +582,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -621,7 +642,7 @@ jobs:
PYBUILD
- name: Upload debug binary
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-debug-binary
path: |
@@ -646,7 +667,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -662,7 +683,7 @@ jobs:
run: python3 scripts/e2e_binary.py build --bins --features rio-v2,e2e-test-hooks
- name: Upload debug binary
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-debug-binary-rio-v2
path: |
@@ -690,7 +711,7 @@ jobs:
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -754,7 +775,7 @@ jobs:
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -770,7 +791,7 @@ jobs:
install-build-packaging-tools: 'false'
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -785,7 +806,7 @@ jobs:
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
@@ -821,7 +842,7 @@ jobs:
- name: Upload e2e smoke diagnostics
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-diagnostics-${{ github.run_number }}
path: |
@@ -831,7 +852,7 @@ jobs:
- name: Upload e2e smoke JUnit report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-junit-${{ github.run_number }}
path: target/nextest/e2e-smoke/junit.xml
@@ -856,7 +877,7 @@ jobs:
- name: Upload test logs
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-test-logs-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
@@ -879,7 +900,7 @@ jobs:
timeout-minutes: 55
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -903,7 +924,7 @@ jobs:
fi
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -945,7 +966,7 @@ jobs:
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
@@ -1001,11 +1022,12 @@ jobs:
RUSTFS_E2E_STARTUP_CAS_BINARY: ${{ runner.temp }}/rustfs-startup-cas-input/rustfs
RUSTFS_E2E_STARTUP_CAS_BUILD_MANIFEST: ${{ runner.temp }}/rustfs-startup-cas-input/rustfs.e2e-startup-cas-build.json
RUSTFS_E2E_STARTUP_CAS_ARTIFACT_DIR: ${{ runner.temp }}/rustfs-startup-cas-evidence
RUSTFS_HEAL_CHAOS_LOG_ROOT: ${{ runner.temp }}/rustfs-heal-chaos-logs
run: python3 scripts/e2e_binary.py run --binary "$RUSTFS_E2E_STARTUP_CAS_BINARY" --features e2e-test-hooks -- cargo nextest run --profile e2e-full -p e2e_test
- name: Upload junit
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-full-junit-${{ github.run_number }}
path: |
@@ -1015,7 +1037,7 @@ jobs:
- name: Upload startup CAS evidence
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: fresh-startup-cas-evidence-${{ github.run_number }}
path: |
@@ -1024,6 +1046,15 @@ jobs:
if-no-files-found: warn
retention-days: 7
- name: Upload heal cluster logs
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-full-heal-logs-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-heal-chaos-logs
if-no-files-found: warn
retention-days: 7
e2e-tests-rio-v2:
name: End-to-End Tests (rio-v2)
# Inherits the schedule/dispatch-only gate through needs: on every other
@@ -1034,7 +1065,7 @@ jobs:
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -1044,7 +1075,7 @@ jobs:
rm -f /tmp/rustfs.log
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary-rio-v2
path: target/debug
@@ -1077,7 +1108,7 @@ jobs:
- name: Upload test logs
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-test-logs-rio-v2-${{ github.run_number }}
path: /tmp/rustfs.log
@@ -1090,12 +1121,12 @@ jobs:
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
@@ -1120,7 +1151,7 @@ jobs:
- name: Upload s3 test artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: s3tests-implemented-${{ github.run_number }}
path: artifacts/s3tests-single/**
@@ -1164,12 +1195,12 @@ jobs:
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download debug binary
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
@@ -1202,7 +1233,7 @@ jobs:
- name: Upload s3 test artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: s3tests-lifecycle-behavior-${{ github.run_number }}
path: artifacts/s3tests-single/**
@@ -1230,10 +1261,10 @@ jobs:
- build-rustfs-debug-binary-rio-v2
- e2e-tests-rio-v2
- e2e-full
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Require the expected result of every CI lane
@@ -1272,7 +1303,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+3 -3
View File
@@ -46,7 +46,7 @@ jobs:
# Echoes one line; the run exists only so the concurrency group cancels the
# in-flight run of a closed PR.
permissions: {}
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Explain cancellation run
@@ -61,12 +61,12 @@ jobs:
checks: write
issues: write
pull-requests: write
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
steps:
- name: Report CLA result for merge queue
if: github.event_name == 'merge_group'
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
await github.rest.checks.create({
@@ -0,0 +1,198 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Connect diagnostic scheduler receipt acceptance
on:
workflow_dispatch:
inputs:
build_run_id:
description: Main-branch Build and Release run ID
required: true
type: string
artifact_id:
description: Linux x86_64 GNU artifact from that run
required: true
type: string
source_sha:
description: Exact RustFS source commit
required: true
type: string
artifact_digest:
description: GitHub artifact digest including sha256 prefix
required: true
type: string
binary_sha256:
description: Independently verified rustfs binary SHA-256
required: true
type: string
connect_sha:
description: Exact Connect scheduler acceptance harness commit
required: true
type: string
permissions:
actions: read
contents: read
jobs:
diagnostic-schedule:
name: Verify production scheduler receipts over mTLS
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/connect
path: connect-harness
persist-credentials: false
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Verify official source and artifact identity
shell: bash
env:
GH_TOKEN: ${{ github.token }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
SOURCE_SHA: ${{ inputs.source_sha }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
run: |
set -euo pipefail
[[ "$BUILD_RUN_ID" =~ ^[0-9]+$ && "$ARTIFACT_ID" =~ ^[0-9]+$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
[[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
jq -e --arg source "$SOURCE_SHA" --arg repository "$GITHUB_REPOSITORY" '
.head_sha == $source and .head_branch == "main"
and .head_repository.full_name == $repository
and .name == "Build and Release" and .path == ".github/workflows/build.yml"
and .status == "completed" and .conclusion == "success"
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
.workflow_run.id == $run and .workflow_run.head_sha == $source
and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
and .digest == $digest and .expired == false
and (.expires_at | fromdateiso8601) > now
and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
' <<<"$artifact" >/dev/null
gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
printf '%s official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
- name: Extract the verified official binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
run: |
set -euo pipefail
python3 - <<'PY'
import io
import os
import pathlib
import stat
import zipfile
def regular(entry):
mode = entry.external_attr >> 16
return not entry.is_dir() and (stat.S_IFMT(mode) in (0, stat.S_IFREG))
package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
with zipfile.ZipFile('official-artifact.zip') as outer:
entries = outer.infolist()
assert 1 <= len(entries) <= 16
assert len({item.filename for item in entries}) == len(entries)
assert all(regular(item) and pathlib.PurePosixPath(item.filename).name == item.filename for item in entries)
assert sum(item.file_size for item in entries) <= 2147483648
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
assert {item.filename for item in entries} == {'rustfs', 'rustfs-cli'} and len(entries) == 2
assert all(regular(item) and 0 < item.file_size <= 1073741824 for item in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
import shutil
shutil.copyfileobj(source, destination)
PY
chmod 0755 binary/rustfs
printf '%s binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
file binary/rustfs | grep -Eq 'ELF 64-bit LSB.*x86-64'
- name: Run the real scheduler and receipt delivery scenario
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
git -C connect-harness diff --exit-code
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
[[ $(git -C connect-harness diff --name-only) == tests/e2e/connected/rustfs-ref ]]
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/diagnostic-schedule-evidence.test.sh)
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_DIAGNOSTIC_SCHEDULE_EVIDENCE="$GITHUB_WORKSPACE/diagnostic-schedule-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=diagnostic-schedule
- name: Bind and validate sanitized evidence
shell: bash
env:
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
jq -e --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" '
.provenance.connectSha == $connect and .provenance.sourceSha == $source
and .provenance.binarySha256 == $binary
' diagnostic-schedule-evidence.json >/dev/null
jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
--arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" '
.provenance += {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest}
' diagnostic-schedule-evidence.json > diagnostic-schedule-evidence.bound.json
mv diagnostic-schedule-evidence.bound.json diagnostic-schedule-evidence.json
node connect-harness/tests/connectivity/verify-diagnostic-schedule-evidence.mjs diagnostic-schedule-evidence.json --bound
- name: Upload verified receipt evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: connect-diagnostic-schedule-evidence-${{ github.run_id }}
path: diagnostic-schedule-evidence.json
retention-days: 14
if-no-files-found: error
@@ -0,0 +1,236 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Connect bounded health service acceptance
on:
workflow_dispatch:
inputs:
build_run_id:
description: Successful main-branch Build and Release run ID
required: true
type: string
artifact_id:
description: Linux x86_64 GNU artifact from that run
required: true
type: string
source_sha:
description: Exact RustFS source commit
required: true
type: string
artifact_digest:
description: GitHub artifact digest including sha256 prefix
required: true
type: string
binary_sha256:
description: Independently verified rustfs binary SHA-256
required: true
type: string
connect_sha:
description: Exact Connect health service acceptance harness commit
required: true
type: string
permissions:
actions: read
contents: read
jobs:
health-service:
name: Verify bounded health service over mTLS
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/connect
path: connect-harness
persist-credentials: false
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Verify official source and artifact identity
shell: bash
env:
GH_TOKEN: ${{ github.token }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
SOURCE_SHA: ${{ inputs.source_sha }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
run: |
set -euo pipefail
[[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
[[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
jq -e --arg source "$SOURCE_SHA" '
.head_sha == $source and .head_branch == "main"
and .head_repository.full_name == "rustfs/rustfs"
and .name == "Build and Release" and .path == ".github/workflows/build.yml"
and .status == "completed" and .conclusion == "success"
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
.workflow_run.id == $run and .workflow_run.head_sha == $source
and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
and .digest == $digest and .expired == false
and (.expires_at | fromdateiso8601) > now
and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
' <<<"$artifact" >/dev/null
jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
[[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
printf '%s official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
- name: Extract only the verified official service binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
run: |
set -euo pipefail
python3 - <<'PY'
import io
import os
import pathlib
import shutil
import stat
import zipfile
def regular(entry):
mode = entry.external_attr >> 16
return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
with zipfile.ZipFile('official-artifact.zip') as outer:
entries = outer.infolist()
assert 1 <= len(entries) <= 16
assert len({entry.filename for entry in entries}) == len(entries)
assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
assert sum(entry.file_size for entry in entries) <= 2147483648
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
assert {entry.filename for entry in entries} == {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 4
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
header = source.read(20)
assert header[:7] == b'\x7fELF\x02\x01\x01' and header[18:20] == b'\x3e\x00', 'Expected a little-endian ELF64 x86-64 binary'
destination.write(header)
shutil.copyfileobj(source, destination)
PY
chmod 0755 binary/rustfs
printf '%s binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
- name: Run the real health service job
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
git -C connect-harness diff --exit-code
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
changed=$(git -C connect-harness diff --name-only)
[[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh)
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=health-service
- name: Bind and validate sanitized evidence
shell: bash
env:
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
jq -e --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" '
.provenance.connectSha == $connect and .provenance.sourceSha == $source
and .provenance.binarySha256 == $binary
' health-service-evidence.json >/dev/null
jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
--arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
--arg expiry "$(cat artifact-expires-at)" '
.provenance += {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry}
' health-service-evidence.json > health-service-evidence.bound.json
mv health-service-evidence.bound.json health-service-evidence.json
node connect-harness/tests/connectivity/health-service-evidence.mjs health-service-evidence.json --bound
- name: Upload only verified sanitized evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: connect-health-service-evidence-${{ github.run_id }}
path: health-service-evidence.json
retention-days: 14
if-no-files-found: error
@@ -49,23 +49,23 @@ permissions:
jobs:
performance-drive:
name: Verify native Linux x86_64 performance.drive service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,7 +76,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +122,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -183,7 +183,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('performance-drive-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-performance-drive-evidence-${{ github.run_id }}
path: |
@@ -49,23 +49,23 @@ permissions:
jobs:
performance-network:
name: Verify native Linux x86_64 performance.network service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,7 +76,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +122,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -183,7 +183,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('performance-network-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-performance-network-evidence-${{ github.run_id }}
path: |
@@ -37,6 +37,10 @@ on:
description: Expected rustfs binary SHA-256
required: true
type: string
catalog_sha256:
description: Independently reviewed release CPU catalog SHA-256
required: true
type: string
connect_sha:
description: Exact 40-character RustFS Connect harness commit
required: true
@@ -49,23 +53,23 @@ permissions:
jobs:
profile-cpu:
name: Verify native Linux x86_64 profile.cpu service job
runs-on: ubuntu-latest
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,7 +80,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -101,20 +105,27 @@ jobs:
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
github_api() {
curl --fail --silent --show-error \
--header "Authorization: Bearer $GH_TOKEN" \
--header 'Accept: application/vnd.github+json' \
"https://api.github.com/$1"
}
run=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
[[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]]
[[ $(jq -r '.head_branch' <<<"$run") == main ]]
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
[.jobs[] | select(.name == $name)] as $matches
| (($matches | length) == 1 and $matches[0].conclusion == "success")
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
artifact=$(github_api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
[[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]]
[[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]]
[[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]]
@@ -122,31 +133,39 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
run-id: ${{ inputs.build_run_id }}
github-token: ${{ github.token }}
- name: Extract exact RustFS binary
- name: Extract exact RustFS binary and reviewed catalog
shell: bash
env:
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CATALOG_SHA256: ${{ inputs.catalog_sha256 }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ "$CATALOG_SHA256" =~ ^[0-9a-f]{64}$ ]]
short_sha=${SOURCE_SHA:0:7}
package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit)
[[ -n "$package" ]]
mkdir -p binary
unzip -qq "$package" rustfs -d binary
unzip -qq "$package" rustfs rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256 -d binary
chmod +x binary/rustfs
printf '%s %s\n' "$BINARY_SHA256" binary/rustfs | sha256sum --check --strict
# The packaged digest is corroboration, never the independent trust input.
[[ $(cat binary/rustfs.cpu-symbol-catalog.sha256) == "$CATALOG_SHA256" ]]
printf '%s %s\n' "$CATALOG_SHA256" binary/rustfs.cpu-symbol-catalog.json | sha256sum --check --strict
chmod 0444 binary/rustfs.cpu-symbol-catalog.json
- name: Run signed service job acceptance
shell: bash
env:
CATALOG_SHA256: ${{ inputs.catalog_sha256 }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
@@ -156,6 +175,8 @@ jobs:
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \
CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \
CONNECT_E2E_PROFILE_EVIDENCE="$GITHUB_WORKSPACE/profile-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=profile
@@ -183,7 +204,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('profile-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-profile-cpu-evidence-${{ github.run_id }}
path: |
@@ -0,0 +1,235 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Connect service memory profile acceptance
on:
workflow_dispatch:
inputs:
build_run_id:
description: Successful main-branch Build and Release run ID
required: true
type: string
artifact_id:
description: Linux x86_64 GNU artifact from that run
required: true
type: string
source_sha:
description: Exact RustFS source commit
required: true
type: string
artifact_digest:
description: GitHub artifact digest including sha256 prefix
required: true
type: string
binary_sha256:
description: Independently verified rustfs binary SHA-256
required: true
type: string
connect_sha:
description: Exact Connect memory acceptance harness commit
required: true
type: string
permissions:
actions: read
contents: read
jobs:
profile-memory:
name: Verify signed service memory profile over mTLS
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/connect
path: connect-harness
persist-credentials: false
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Verify official source and artifact identity
shell: bash
env:
GH_TOKEN: ${{ github.token }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
SOURCE_SHA: ${{ inputs.source_sha }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
run: |
set -euo pipefail
[[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
[[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
jq -e --arg source "$SOURCE_SHA" '
.head_sha == $source and .head_branch == "main"
and .head_repository.full_name == "rustfs/rustfs"
and .name == "Build and Release" and .path == ".github/workflows/build.yml"
and .status == "completed" and .conclusion == "success"
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
.workflow_run.id == $run and .workflow_run.head_sha == $source
and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
and .digest == $digest and .expired == false
and (.expires_at | fromdateiso8601) > now
and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
' <<<"$artifact" >/dev/null
jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
[[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
printf '%s official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
- name: Extract only the verified official service binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
run: |
set -euo pipefail
python3 - <<'PY'
import io
import os
import pathlib
import shutil
import stat
import zipfile
def regular(entry):
mode = entry.external_attr >> 16
return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
with zipfile.ZipFile('official-artifact.zip') as outer:
entries = outer.infolist()
assert 1 <= len(entries) <= 16
assert len({entry.filename for entry in entries}) == len(entries)
assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
assert sum(entry.file_size for entry in entries) <= 2147483648
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
shutil.copyfileobj(source, destination)
PY
chmod 0755 binary/rustfs
printf '%s binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
file binary/rustfs | grep -Eq 'ELF 64-bit LSB.*x86-64'
- name: Run the real service job under S3 workload
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
git -C connect-harness diff --exit-code
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
changed=$(git -C connect-harness diff --name-only)
[[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/profile-memory-evidence.test.sh)
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PROFILE_MEMORY_EVIDENCE="$GITHUB_WORKSPACE/profile-memory-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=profile-memory
- name: Bind and validate sanitized evidence
shell: bash
env:
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
jq -e --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" '
.provenance.connectSha == $connect and .provenance.sourceSha == $source
and .provenance.binarySha256 == $binary
' profile-memory-evidence.json >/dev/null
jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
--arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
--arg expiry "$(cat artifact-expires-at)" '
.provenance += {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry}
' profile-memory-evidence.json > profile-memory-evidence.bound.json
mv profile-memory-evidence.bound.json profile-memory-evidence.json
node connect-harness/tests/connectivity/profile-memory-evidence.mjs profile-memory-evidence.json --bound
- name: Upload only verified sanitized evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: connect-profile-memory-evidence-${{ github.run_id }}
path: profile-memory-evidence.json
retention-days: 14
if-no-files-found: error
@@ -17,6 +17,14 @@ name: Connect profile.threads service-job acceptance
on:
workflow_dispatch:
inputs:
thread_scope:
description: Thread profile scope
required: true
type: choice
default: native
options:
- native
- runtime
build_run_id:
description: Main-branch Build and Release workflow run ID
required: true
@@ -48,24 +56,24 @@ permissions:
jobs:
profile-threads:
name: Verify native Linux x86_64 profile.threads service job
runs-on: ubuntu-latest
name: Verify Linux x86_64 profile.threads service job
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,10 +84,43 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Verify source run and artifact identity
shell: bash
env:
@@ -107,7 +148,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +163,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -148,8 +189,14 @@ jobs:
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
THREAD_SCOPE: ${{ inputs.thread_scope }}
run: |
set -euo pipefail
case "$THREAD_SCOPE" in
native) export CONNECT_E2E_THREAD_SCOPE=NATIVE_THREADS ;;
runtime) export CONNECT_E2E_THREAD_SCOPE=TOKIO_RUNTIME ;;
*) exit 1 ;;
esac
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
@@ -168,6 +215,7 @@ jobs:
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
THREAD_SCOPE: ${{ inputs.thread_scope }}
run: |
set -euo pipefail
jq \
@@ -177,13 +225,14 @@ jobs:
--arg artifactDigest "$ARTIFACT_DIGEST" \
--arg connectSha "$CONNECT_SHA" \
--arg sourceSha "$SOURCE_SHA" \
'. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha}' \
--arg requestedThreadScope "$THREAD_SCOPE" \
'. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha, requestedThreadScope: $requestedThreadScope}' \
profile-threads-service-job-evidence.json >profile-threads-service-job-evidence.bound.json
mv profile-threads-service-job-evidence.bound.json profile-threads-service-job-evidence.json
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('profile-threads-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-profile-threads-evidence-${{ github.run_id }}
path: |
@@ -49,23 +49,23 @@ permissions:
jobs:
top-api:
name: Verify native Linux x86_64 top.api service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,7 +76,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +122,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -183,7 +183,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('top-api-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-top-api-evidence-${{ github.run_id }}
path: |
+177 -68
View File
@@ -12,21 +12,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.
name: Connect top.disk artifact acceptance
name: Connect top.disk service-job acceptance
on:
workflow_dispatch:
inputs:
build_run_id:
description: Successful main-branch Build and Release workflow run ID
description: Successful main-branch Build and Release run ID
required: true
type: string
artifact_id:
description: Linux x86_64 GNU artifact ID from that run
description: Linux x86_64 GNU artifact from that run
required: true
type: string
source_sha:
description: Exact 40-character source commit
description: Exact RustFS source commit
required: true
type: string
artifact_digest:
@@ -34,7 +34,11 @@ on:
required: true
type: string
binary_sha256:
description: Expected rustfs binary SHA-256
description: Independently verified rustfs binary SHA-256
required: true
type: string
connect_sha:
description: Exact Connect top.disk service-job acceptance harness commit
required: true
type: string
@@ -44,16 +48,67 @@ permissions:
jobs:
top-disk:
name: Verify native Linux x86_64 top.disk artifact
runs-on: ubuntu-latest
timeout-minutes: 15
name: Verify native top.disk service job over mTLS
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Checkout exact RustFS source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Verify source run and artifact identity
- name: Checkout exact Connect harness
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
persist-credentials: false
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Verify official source and artifact identity
shell: bash
env:
GH_TOKEN: ${{ github.token }}
@@ -61,82 +116,136 @@ jobs:
ARTIFACT_ID: ${{ inputs.artifact_id }}
SOURCE_SHA: ${{ inputs.source_sha }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
run: |
set -euo pipefail
[[ "$BUILD_RUN_ID" =~ ^[0-9]+$ ]]
[[ "$ARTIFACT_ID" =~ ^[0-9]+$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
[[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
jq -e --arg source "$SOURCE_SHA" '
.head_sha == $source and .head_branch == "main"
and .head_repository.full_name == "rustfs/rustfs"
and .name == "Build and Release" and .path == ".github/workflows/build.yml"
and .status == "completed" and .conclusion == "success"
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu, [a-z0-9-]+, x86_64-unknown-linux-gnu, false, linux, pyroscope\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
.workflow_run.id == $run and .workflow_run.head_sha == $source
and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
and .digest == $digest and .expired == false
and (.expires_at | fromdateiso8601) > now
and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
' <<<"$artifact" >/dev/null
jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
[[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
printf '%s official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
[[ $(jq -r '.conclusion' <<<"$run") == success ]]
[[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]]
[[ $(jq -r '.head_branch' <<<"$run") == main ]]
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
[[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]]
[[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]]
[[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]]
[[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]]
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
run-id: ${{ inputs.build_run_id }}
github-token: ${{ github.token }}
- name: Extract exact RustFS binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
short_sha=${SOURCE_SHA:0:7}
package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit)
[[ -n "$package" ]]
mkdir -p binary
unzip -qq "$package" rustfs -d binary
chmod +x binary/rustfs
- name: Run top.disk acceptance
- name: Extract only the verified official service binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
run: |
set -euo pipefail
scripts/ci/check_connect_top_disk_artifact.sh \
binary/rustfs "$SOURCE_SHA" "$BINARY_SHA256" \
top-disk-runtime-evidence.json top-disk.zip
python3 - <<'PY'
import io
import os
import pathlib
import shutil
import stat
import zipfile
- name: Bind workflow and artifact provenance
def regular(entry):
mode = entry.external_attr >> 16
return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
with zipfile.ZipFile('official-artifact.zip') as outer:
entries = outer.infolist()
assert 1 <= len(entries) <= 16
assert len({entry.filename for entry in entries}) == len(entries)
assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
assert sum(entry.file_size for entry in entries) <= 2147483648
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
expected = {'rustfs', 'rustfs-cli', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
header = source.read(20)
assert header[:7] == b'\x7fELF\x02\x01\x01' and header[18:20] == b'\x3e\x00', 'Expected a little-endian ELF64 x86-64 binary'
destination.write(header)
shutil.copyfileobj(source, destination)
PY
chmod 0755 binary/rustfs
printf '%s binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
- name: Run the real top.disk service job
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
CONNECT_E2E_PRODUCER_SOURCE: ${{ inputs.source_sha }}
CONNECT_E2E_PRODUCER_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_E2E_PRODUCER_ARTIFACT_ID: ${{ inputs.artifact_id }}
run: |
set -euo pipefail
git -C connect-harness diff --exit-code
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
changed=$(git -C connect-harness diff --name-only)
[[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_DISK_EVIDENCE="$GITHUB_WORKSPACE/top-disk-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-disk
- name: Bind and validate sanitized evidence
shell: bash
env:
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
jq \
--arg workflowRunId "$GITHUB_RUN_ID" \
--arg buildRunId "$BUILD_RUN_ID" \
--arg artifactId "$ARTIFACT_ID" \
--arg artifactDigest "$ARTIFACT_DIGEST" \
'. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest}' \
top-disk-runtime-evidence.json >top-disk-runtime-evidence.bound.json
mv top-disk-runtime-evidence.bound.json top-disk-runtime-evidence.json
jq -e --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" --arg artifact "$ARTIFACT_ID" '
.status == "PASS" and .officialArtifactId == $artifact and .producerSource == $source and .producerSha256 == $binary
and .result.provenance.sourceCommit == $source
and .result.provenance.executableSha256 == $binary
' top-disk-service-job-evidence.json >/dev/null
jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
--arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
--arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" \
--arg expiry "$(cat artifact-expires-at)" '
. + {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry,connectSha:$connect,sourceSha:$source,binarySha256:$binary}
' top-disk-service-job-evidence.json > top-disk-service-job-evidence.bound.json
mv top-disk-service-job-evidence.bound.json top-disk-service-job-evidence.json
- name: Upload acceptance evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
- name: Upload only verified sanitized evidence
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-top-disk-evidence-${{ github.run_id }}
path: |
top-disk-runtime-evidence.json
top-disk.zip
name: connect-top-disk-service-job-evidence-${{ github.run_id }}
path: top-disk-service-job-evidence.json
retention-days: 14
if-no-files-found: error
@@ -49,23 +49,23 @@ permissions:
jobs:
top-locks:
name: Verify native Linux x86_64 top.locks service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,7 +76,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +122,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -183,7 +183,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('top-locks-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-top-locks-evidence-${{ github.run_id }}
path: |
@@ -49,23 +49,23 @@ permissions:
jobs:
top-rpc:
name: Verify native Linux x86_64 top.rpc service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -76,7 +76,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +107,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-4, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +122,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -183,7 +183,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('top-rpc-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-top-rpc-evidence-${{ github.run_id }}
path: |
+4 -4
View File
@@ -76,7 +76,7 @@ jobs:
NEXTEST_PROFILE: ci
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -89,7 +89,7 @@ jobs:
install-build-packaging-tools: 'false'
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
with:
tool: cargo-llvm-cov
@@ -112,7 +112,7 @@ jobs:
- name: Upload coverage artifact
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: coverage-lcov-${{ github.run_number }}
path: |
@@ -134,7 +134,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+74 -13
View File
@@ -47,7 +47,7 @@ on:
default: true
type: boolean
version:
description: "Version to build (latest for stable release, or specific version like v1.0.0, v1.0.0-alpha1)"
description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
required: false
default: "latest"
type: string
@@ -82,9 +82,8 @@ jobs:
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main' &&
!contains(github.event.workflow_run.head_branch, '-preview'))
runs-on: ubuntu-latest
github.event.workflow_run.head_branch != 'main')
runs-on: sm-standard-2
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
@@ -97,7 +96,7 @@ jobs:
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
@@ -197,7 +196,14 @@ jobs:
version="${version#v}"
fi
if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Building Docker image for preview: $version (version tags only)"
elif [[ "$version" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
@@ -249,11 +255,14 @@ jobs:
echo "🚀 Building with latest stable release version"
;;
*-preview*)
if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
echo "❌ Invalid preview version: $input_version" >&2
exit 1
fi
build_type="preview"
is_prerelease=true
should_build=false
should_push=false
echo "⏭️ Preview tags do not publish Docker images"
version="${input_version#v}"
echo "🔍 Building with preview version: $version (version tags only)"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
@@ -343,7 +352,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
@@ -492,7 +501,7 @@ jobs:
name: Scan Docker Images
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
permissions:
contents: read
@@ -535,19 +544,67 @@ jobs:
category: container-image-${{ matrix.variant }}
- name: Upload container scan report
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: container-image-scan-${{ matrix.variant }}
path: trivy-${{ matrix.variant }}.sarif
if-no-files-found: error
retention-days: 30
# Docker Hub's overview is a separate `full_description` field that
# `docker push` never touches, so it drifted from README.md and kept
# linking to pages that no longer exist (#7995). Republish it from the
# commit the images were built from, only when images were pushed.
sync-dockerhub-description:
name: Sync Docker Hub Description
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
sparse-checkout: README.md
sparse-checkout-cone-mode: false
- name: Sync Docker Hub description
uses: peter-evans/dockerhub-description@1b9a80c056b620d92cedb9d9b5a223409c68ddfa # v5.0.0
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
repository: ${{ env.REGISTRY_DOCKERHUB }}
readme-filepath: ./README.md
# README.md links to docs/ and CONTRIBUTING.md by relative path;
# rewrite those to github.com URLs so they resolve on Docker Hub.
enable-url-completion: true
# Docker Hub caps `full_description` at 25,000 bytes and the action
# truncates to fit with only a warning, which would publish a README
# cut off mid-sentence. README.md is already near the cap, so read the
# published overview back and fail if it hit the limit.
- name: Check published description was not truncated
env:
REPOSITORY: ${{ env.REGISTRY_DOCKERHUB }}
run: |
limit=25000
published=$(curl -sSf "https://hub.docker.com/v2/repositories/${REPOSITORY}/" \
| jq -j '.full_description' | wc -c | tr -d ' ')
echo "Published overview: ${published} bytes (Docker Hub limit: ${limit})"
# Truncation lands within one UTF-8 character of the cap.
if [ "$published" -ge $((limit - 4)) ]; then
echo "::error file=README.md::Docker Hub overview was truncated at ${limit} bytes; trim README.md before the next release."
exit 1
fi
# Docker build summary
docker-summary:
name: Docker Build Summary
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Docker build completion summary
@@ -563,6 +620,10 @@ jobs:
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview Docker image has been built with ${VERSION} tags"
echo "⏭️ Preview images do not update latest or prerelease channels"
;;
"release")
echo "🚀 Release Docker image has been built with ${VERSION} tags"
echo "✅ This image is ready for production use"
+4 -3
View File
@@ -38,6 +38,7 @@ on:
- "Cargo.lock"
- "Cargo.toml"
- ".config/nextest.toml"
- ".config/e2e-distributed-selection.txt"
- ".github/workflows/e2e-distributed.yml"
- "crates/audit/**"
- "crates/common/**"
@@ -93,7 +94,7 @@ jobs:
UPGRADE_SOURCE_SHA256: 7c789386bf85278f865b8e0d359bf4edb84d5aa408cc3fa54a18c25ca74cd6e7
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -174,7 +175,7 @@ jobs:
- name: Upload distributed e2e diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-distributed-${{ github.run_number }}
path: |
@@ -206,7 +207,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
@@ -63,7 +63,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -77,7 +77,7 @@ jobs:
# The STS dual-node test requires awscurl and fails if it is unavailable.
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -108,7 +108,7 @@ jobs:
- name: Upload nextest junit report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-replication-nightly-junit-${{ github.run_number }}
path: |
@@ -126,7 +126,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -157,7 +157,7 @@ jobs:
- name: Upload cluster fault diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-cluster-nightly-${{ github.run_number }}
path: |
@@ -176,7 +176,7 @@ jobs:
RUSTFS_BUILD_FEATURES: ftps,webdav,sftp
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -213,7 +213,7 @@ jobs:
- name: Upload protocol diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-protocol-nightly-${{ github.run_number }}
path: |
@@ -236,7 +236,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+14 -9
View File
@@ -159,7 +159,7 @@ jobs:
TEST_MODE: ${{ matrix.test-mode }}
S3_SHARD_INDEX: ${{ matrix.shard-index }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -167,7 +167,7 @@ jobs:
# ship a working pip (ci-1: a bare python3 without pip is what broke the
# scheduled sweep). Keeps the workflow correct across runner drift.
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -264,7 +264,7 @@ jobs:
hostname: rustfs4
volumes: [rustfs4-data:/data]
lb:
image: haproxy:2.9
image: haproxy:3.4.4
hostname: lb
networks: [rustfs-net]
ports:
@@ -308,6 +308,11 @@ jobs:
docker compose -f compose.yml up -d
# Validate the pinned HAProxy image parses the exact config before
# the S3 compatibility sweep starts.
docker compose -f compose.yml exec -T lb \
haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg
- name: Wait for RustFS ready
run: |
for _ in {1..120}; do
@@ -357,7 +362,7 @@ jobs:
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: s3tests-${{ env.TEST_MODE }}-shard-${{ matrix.shard-index }}
path: artifacts/**
@@ -366,15 +371,15 @@ jobs:
name: Upstream HEAD classification canary
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
continue-on-error: true
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 20
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -435,7 +440,7 @@ jobs:
- name: Upload canary artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: s3tests-upstream-head
path: artifacts/s3tests-upstream-head/**
@@ -454,7 +459,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+66 -21
View File
@@ -51,8 +51,53 @@ env:
UPGRADE_SOURCE_SHA256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d
jobs:
build:
name: Build upgrade candidate
runs-on: sm-standard-2
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-shared-key: e2e-upgrade-server
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Download pinned previous release
run: |
mkdir -p target/debug
archive="target/debug/$UPGRADE_SOURCE_ASSET"
curl --fail --location --retry 3 --output "$archive" \
"https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
echo "$UPGRADE_SOURCE_SHA256 $archive" | sha256sum --check --strict
- name: Build current RustFS binary
run: python3 scripts/e2e_binary.py build
- name: Upload verified upgrade candidate
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: upgrade-candidate-${{ github.run_id }}
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
target/debug/${{ env.UPGRADE_SOURCE_ASSET }}
if-no-files-found: error
retention-days: 3
compression-level: 0
overwrite: true
upgrade:
name: ${{ matrix.name }}
needs: build
strategy:
fail-fast: false
matrix:
@@ -63,59 +108,61 @@ jobs:
# the CI required-check names. UPGRADE_SOURCE_VERSION above is the
# single source of truth for which release they actually run against.
- name: Direct upgrade from the previous release
cache_key: e2e-direct-upgrade
test: direct_upgrade_from_rc2_preserves_object_contracts
artifact: direct-upgrade
- name: Mixed-version rolling upgrade from the previous release
cache_key: e2e-mixed-version-upgrade
test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
artifact: mixed-version-upgrade
- name: Bucket configuration survives the upgrade
cache_key: e2e-bucket-config-upgrade
test: direct_upgrade_from_previous_release_preserves_bucket_configuration
artifact: bucket-config-upgrade
- name: Rollback reads current bucket metadata
cache_key: e2e-bucket-config-rollback
test: rollback_to_previous_release_reads_current_bucket_metadata
artifact: bucket-config-rollback
- name: ODM configuration recovery after rc.5 rollback
cache_key: e2e-odm-config-rollback
test: rc5_rollback_requires_restoring_odm_configuration
artifact: odm-config-rollback
- name: Multipart layouts survive the rc.5 upgrade
cache_key: e2e-multipart-layout-upgrade
test: direct_upgrade_from_rc5_preserves_multipart_layouts
artifact: multipart-layout-upgrade
- name: rc.5 multipart replication baseline
cache_key: e2e-multipart-layout-baseline
test: rc5_baseline_replicates_multipart_layouts
artifact: multipart-layout-baseline
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-shared-key: ${{ matrix.cache_key }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
cache-shared-key: e2e-upgrade-tests
# One main-branch consumer warms the shared test-harness dependencies.
cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Download pinned previous release
- name: Download verified upgrade candidate
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: upgrade-candidate-${{ github.run_id }}
path: target/debug
- name: Restore candidate executable bit
run: chmod +x target/debug/rustfs
- name: Verify and unpack pinned previous release
env:
SOURCE_DIR: ${{ runner.temp }}/rustfs-upgrade-source
run: |
set -euo pipefail
mkdir -p "$SOURCE_DIR"
archive="$SOURCE_DIR/$UPGRADE_SOURCE_ASSET"
curl --fail --location --retry 3 --output "$archive" \
"https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
archive="target/debug/$UPGRADE_SOURCE_ASSET"
echo "$UPGRADE_SOURCE_SHA256 $archive" | sha256sum --check --strict
unzip -q "$archive" -d "$SOURCE_DIR"
chmod +x "$SOURCE_DIR/rustfs"
@@ -123,21 +170,19 @@ jobs:
echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
- name: Build current RustFS binary
run: |
python3 scripts/e2e_binary.py build
- name: Run upgrade compatibility test
env:
RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
NO_PROXY: 127.0.0.1,localhost,::1
run: |
export no_proxy="$NO_PROXY"
python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
"upgrade_compatibility_test::${{ matrix.test }}" \
-- --ignored --exact --nocapture
- name: Upload scanner/heal G09 evidence
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
@@ -146,7 +191,7 @@ jobs:
- name: Upload server logs
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-logs
+35 -3
View File
@@ -11,12 +11,44 @@ concurrency:
cancel-in-progress: false
jobs:
collect:
runs-on: ubuntu-latest
runs-on: smoke-testing
timeout-minutes: 15
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Collect and publish verified chain health
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
@@ -24,7 +56,7 @@ jobs:
run: python3 scripts/functional_chain_health.py --source-ref "${NIGHTLY_SOURCE_REF}" --publish --output "${RUNNER_TEMP}/chain-health.json"
- name: Retain health observation
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-health-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-health.json
+11 -11
View File
@@ -60,7 +60,7 @@ jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Explain cancellation run
@@ -81,7 +81,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -93,7 +93,7 @@ jobs:
cache-save-if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }}
- name: Install cargo-fuzz
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
with:
tool: cargo-fuzz
@@ -113,7 +113,7 @@ jobs:
done
- name: Upload prebuilt fuzz binaries
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: fuzz-prebuilt-binaries-${{ github.run_number }}
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/
@@ -143,12 +143,12 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: fuzz-prebuilt-binaries-${{ github.run_number }}
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
@@ -167,7 +167,7 @@ jobs:
- name: Upload fuzz smoke artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: fuzz-smoke-${{ matrix.target }}-${{ github.run_number }}
path: |
@@ -197,12 +197,12 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: fuzz-prebuilt-binaries-${{ github.run_number }}
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
@@ -221,7 +221,7 @@ jobs:
- name: Upload nightly fuzz artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: fuzz-nightly-${{ matrix.target }}-${{ github.run_number }}
path: |
@@ -246,7 +246,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+42 -7
View File
@@ -31,7 +31,7 @@ permissions:
jobs:
build-helm-package:
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
@@ -48,9 +48,10 @@ jobs:
chart_version: ${{ steps.version.outputs.chart_version }}
steps:
- name: Checkout helm chart repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Checkout release source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
persist-credentials: false
# Both inputs reach the shell through env rather than `${{ }}`
@@ -83,6 +84,40 @@ jobs:
./scripts/helm_chart_version.sh "$RAW_TAG"
# Docker and Helm share the binary-build trigger. Do not publish a chart
# until the image selected by its default appVersion is pullable.
- name: Wait for release image
timeout-minutes: 20
env:
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -euo pipefail
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
[[ "$APP_VERSION" == *-preview* ]]; then
echo "Invalid deliverable image version: $APP_VERSION" >&2
exit 1
fi
for attempt in {1..60}; do
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
| jq -er '.token // empty') &&
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
--header "Authorization: Bearer $token" \
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
exit 0
fi
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
if (( attempt < 60 )); then
sleep 15
fi
done
echo "Release image is unavailable; refusing to publish Helm chart" >&2
exit 1
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
@@ -107,21 +142,21 @@ jobs:
--version "${{ steps.version.outputs.chart_version }}"
- name: Upload helm package as artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: helm-package
path: helm/rustfs/*.tgz
retention-days: 1
publish-helm-package:
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
steps:
- name: Checkout helm package repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
@@ -129,7 +164,7 @@ jobs:
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: helm-package
path: ./
+1 -1
View File
@@ -32,7 +32,7 @@ permissions:
jobs:
build:
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- uses: usthe/issues-translate-action@b41f55ddc81d7d54bd542a4f289fe28ec081898e # v2.7
+3 -3
View File
@@ -60,7 +60,7 @@ permissions:
jobs:
minio-interop:
name: MinIO interop (EC + SSE read parity)
# Skip on forks: needs the repo's runners and is not a contributor gate.
# Keep this scheduled interoperability lane scoped to the upstream repo.
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 40
@@ -83,7 +83,7 @@ jobs:
INTEROP_REQUIRED_TESTS: '["reads_minio_generated_sse_s3_multipart_fixture", "reads_minio_generated_sse_kms_multipart_fixture", "rejects_minio_generated_sse_s3_fixture_with_wrong_kms_key", "rejects_minio_generated_sse_s3_fixture_with_truncated_ciphertext"]'
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -137,7 +137,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+4 -4
View File
@@ -114,10 +114,10 @@ jobs:
# label was abandoned (no docker.sock on the pod its only run landed on).
# TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
# run) is added by the ci-8 composite action; do not implement it here.
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 120
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -246,7 +246,7 @@ jobs:
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: mint
path: artifacts/**
@@ -264,7 +264,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+41 -74
View File
@@ -44,14 +44,14 @@ env:
jobs:
resolve-source:
name: Resolve nightly source
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
outputs:
source_sha: ${{ steps.source.outputs.sha }}
source_ref: ${{ steps.source.outputs.ref }}
steps:
- name: Checkout selected source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ env.NIGHTLY_BUILD_REF }}
@@ -71,7 +71,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.resolve-source.outputs.source_sha }}
@@ -117,10 +117,15 @@ jobs:
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://rustfs.com/docs/ for more information
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ROOT_USER=""
# RUSTFS_ROOT_PASSWORD=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
# dpkg versions must start with a digit and cannot contain hyphens;
@@ -146,37 +151,13 @@ jobs:
/etc/default/rustfs
CONFFILES
cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
#!/bin/bash
set -e
if ! getent passwd rustfs > /dev/null 2>&1; then
useradd -r -s /bin/false -d /opt/rustfs rustfs
fi
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
POSTINST
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
systemctl stop rustfs
fi
PRERM
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTRM
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
@@ -214,10 +195,15 @@ jobs:
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://rustfs.com/docs/ for more information
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ROOT_USER=""
# RUSTFS_ROOT_PASSWORD=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
@@ -231,35 +217,13 @@ jobs:
--description "High-performance distributed object storage" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--after-install <(cat << 'POSTINST'
#!/bin/bash
set -e
if ! getent passwd rustfs > /dev/null 2>&1; then
useradd -r -s /bin/false -d /opt/rustfs rustfs
fi
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTINST
) \
--before-remove <(cat << 'PRERM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
systemctl stop rustfs
fi
PRERM
) \
--after-remove <(cat << 'POSTRM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTRM
) \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
"rustfs-nightly-${DEB_DATE}/usr/bin/rustfs=/usr/bin/rustfs" \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
@@ -273,12 +237,12 @@ jobs:
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
- name: Upload DEB artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ steps.deb.outputs.deb_file }}
path: ${{ steps.deb.outputs.deb_file }}
- name: Upload RPM artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ steps.rpm.outputs.rpm_file }}
path: ${{ steps.rpm.outputs.rpm_file }}
@@ -364,7 +328,7 @@ jobs:
- name: Upload nightly candidate manifest
if: ${{ steps.publish.outputs.candidate_file != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: nightly-candidate-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ steps.publish.outputs.candidate_file }}
@@ -396,9 +360,12 @@ jobs:
# live scripts (AppRole login, three-node Raft leader failover).
#
# GitHub-hosted ubuntu-latest, deliberately not the self-hosted sm-standard
# fleet: the HA failover script needs a working Docker daemon, and the
# self-hosted fleet is heterogeneous — a docker-dependent workflow has been
# burned by it before (see the banner in e2e-s3tests.yml, rustfs/backlog#1149).
# fleet: both lanes need a working Docker daemon, and the sm-standard-2 pool
# does not provide one. #8112 rerouted these lanes there and both died in
# seconds on "Docker is not available" while the package itself built and
# published fine, which kept the nightly chain from firing (its gate
# requires a successful build run). See also the banner in e2e-s3tests.yml
# and rustfs/backlog#1149.
kms-vault-lane:
needs: resolve-source
name: KMS live Vault lane
@@ -419,7 +386,7 @@ jobs:
NO_PROXY: 127.0.0.1,localhost
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.resolve-source.outputs.source_sha }}
@@ -509,7 +476,7 @@ jobs:
NO_PROXY: 127.0.0.1,localhost
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.resolve-source.outputs.source_sha }}
@@ -537,7 +504,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+6 -6
View File
@@ -33,30 +33,30 @@ concurrency:
jobs:
update-flake:
name: Update flake.lock
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
# persist-credentials-exempt: update-flake-lock pushes the branch and opens
# the PR. It passes FLAKE_UPDATE_TOKEN to create-pull-request itself rather
# than reusing .git/config, but that is unverified — exempt until a
# workflow_dispatch run confirms it (rustfs/backlog#1602).
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@629b284231c2a82554b724e357e47fc6020833c8 # v3
uses: DeterminateSystems/determinate-nix-action@8d87e8d5e5b8a8309d4281094560f127d9a265f1 # v3.22.5
- name: Cache Nix
uses: DeterminateSystems/flakehub-cache-action@1f9a51a2959d3e26c7838c6f3bf9f48acae525ea # v3.20.0
uses: DeterminateSystems/flakehub-cache-action@83282a8aef353db1d659d3bacf4a28b6683de0b3 # v3.22.5
- name: Check Nix flake inputs
uses: DeterminateSystems/flake-checker-action@3164002371bc90729c68af0e24d5aacf20d7c9f6 # v12
uses: DeterminateSystems/flake-checker-action@786422608c7bded2bbc9741ad9f91356842bf520 # v14
- name: Update flake.lock
id: update
uses: DeterminateSystems/update-flake-lock@fd9359ac79d0e912f1b4b947a48470b3e2799b56 # main
uses: DeterminateSystems/update-flake-lock@da03c0f078bc4b2c37ee4f7e072d34bf8f188bb3 # v29
with:
git-author-name: houseme
git-author-email: housemecn@gmail.com
+6 -6
View File
@@ -54,7 +54,7 @@ jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- name: Explain cancellation run
@@ -63,7 +63,7 @@ jobs:
nix-validation:
name: Nix Build & Check
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 60
permissions:
contents: read
@@ -72,12 +72,12 @@ jobs:
NIX_CURL_FLAGS: -A cargo/stable
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@4eea0b33e3d1f02ecfe37cf16e7204c424009606 # v3.21.0
uses: DeterminateSystems/determinate-nix-action@8d87e8d5e5b8a8309d4281094560f127d9a265f1 # v3.22.5
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
extra-conf: |
@@ -86,10 +86,10 @@ jobs:
max-jobs = 1
- name: Cache Nix
uses: DeterminateSystems/flakehub-cache-action@c01e819d047464c3edf6ba778f075952af5a3aa7 # v3.21.0
uses: DeterminateSystems/flakehub-cache-action@83282a8aef353db1d659d3bacf4a28b6683de0b3 # v3.22.5
- name: Check Nix Flake Inputs
uses: DeterminateSystems/flake-checker-action@3164002371bc90729c68af0e24d5aacf20d7c9f6 # v12
uses: DeterminateSystems/flake-checker-action@786422608c7bded2bbc9741ad9f91356842bf520 # v14
with:
fail-mode: true
ignore-missing-flake-lock: false
+8 -3
View File
@@ -57,7 +57,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -72,6 +72,11 @@ jobs:
- name: Build RustFS
run: cargo build --locked -p rustfs --bin rustfs
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install pinned request signer
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
@@ -82,7 +87,7 @@ jobs:
- name: Upload service logs
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: oidc-keycloak-live-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-keycloak-live-*/**/*.log
@@ -101,7 +106,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
@@ -69,8 +69,7 @@ env:
jobs:
minio-source:
name: MinIO source (read-through, list-through, backfill)
# Skip on forks: needs this repository's runners and is not a contributor
# gate.
# Keep this scheduled interoperability lane scoped to the upstream repo.
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 90
@@ -93,7 +92,7 @@ jobs:
NEXTEST_LISTING: ${{ github.workspace }}/artifacts/odm-interop/minio/selection.json
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -108,11 +107,15 @@ jobs:
run: |
set -euo pipefail
mkdir -p artifacts/odm-interop/minio
docker build -t rustfs-odm-interop-minio \
-f crates/rio-v2/tests/minio_fixture_lab/Dockerfile \
crates/rio-v2/tests/minio_fixture_lab
docker run -d --name rustfs-odm-interop-minio \
--entrypoint /usr/local/bin/minio \
-e "MINIO_ROOT_USER=${MINIO_ROOT_USER}" \
-e "MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}" \
-p 9100:9000 \
minio/minio:RELEASE.2025-09-07T16-13-09Z server /data
rustfs-odm-interop-minio server /data
for _ in $(seq 1 120); do
curl -fsS http://127.0.0.1:9100/minio/health/live >/dev/null 2>&1 && break
sleep 1
@@ -166,7 +169,7 @@ jobs:
- name: Upload the MinIO interop report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: odm-interop-minio-${{ github.run_number }}-${{ github.run_attempt }}
path: |
@@ -183,7 +186,7 @@ jobs:
cloud-source:
name: ${{ matrix.provider }} source (three-case minimum)
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
strategy:
fail-fast: false
@@ -237,7 +240,7 @@ jobs:
- name: Checkout repository
if: steps.credentials.outputs.present == 'true'
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -286,7 +289,7 @@ jobs:
- name: Upload the ${{ matrix.provider }} interop report
if: always() && steps.credentials.outputs.present == 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: odm-interop-${{ matrix.provider }}-${{ github.run_number }}-${{ github.run_attempt }}
path: |
@@ -306,7 +309,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+93 -86
View File
@@ -25,9 +25,10 @@
#
# Flow:
# 1. Resolve and validate the selected Build workflow run and source identity
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu) from build artifacts
# 3. Build DEB packages for amd64 and arm64
# 4. Build RPM packages for x86_64 and aarch64
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
# aarch64-musl) from build artifacts
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
@@ -76,7 +77,7 @@ jobs:
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
@@ -224,11 +225,11 @@ jobs:
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Build DEB and RPM packages for each architecture
# Build DEB and RPM packages for each architecture and libc variant
package:
name: Package (${{ matrix.arch }})
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: resolve
runs-on: ubuntu-latest
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
strategy:
fail-fast: false
@@ -236,16 +237,38 @@ jobs:
matrix:
include:
- arch: x86_64
libc: gnu
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
# gnu binaries dynamically link glibc; musl binaries are static
# and run without a libc dependency.
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: aarch64
libc: gnu
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: x86_64
libc: musl
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-musl"
deb_depends: ""
rpm_depends: ""
- arch: aarch64
libc: musl
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-musl"
deb_depends: ""
rpm_depends: ""
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -256,17 +279,17 @@ jobs:
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
DEB_ARCH: ${{ matrix.deb_arch }}
RPM_ARCH: ${{ matrix.rpm_arch }}
ARCH: ${{ matrix.rpm_arch }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
normalized=$(./scripts/release/package_versions.sh \
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$DEB_ARCH" "$RPM_ARCH")
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
- name: Download binary artifact from build run
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
@@ -308,6 +331,8 @@ jobs:
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
DEB_ARCH: ${{ matrix.deb_arch }}
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
DEB_DEPENDS: ${{ matrix.deb_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
@@ -328,10 +353,15 @@ jobs:
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://rustfs.com/docs/ for more information
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ROOT_USER=""
# RUSTFS_ROOT_PASSWORD=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
@@ -340,9 +370,17 @@ jobs:
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
Depends: libc6 (>= 2.31)
EOF
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
if [[ -n "$DEB_DEPENDS" ]]; then
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
fi
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage
Description: High-performance distributed object storage (${LIBC} build)
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
@@ -354,37 +392,13 @@ jobs:
/etc/default/rustfs
CONFFILES
cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
#!/bin/bash
set -e
if ! getent passwd rustfs > /dev/null 2>&1; then
useradd -r -s /bin/false -d /opt/rustfs rustfs
fi
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
POSTINST
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
systemctl stop rustfs
fi
PRERM
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTRM
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
@@ -408,24 +422,38 @@ jobs:
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
RPM_ARCH: ${{ matrix.rpm_arch }}
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
RPM_DEPENDS: ${{ matrix.rpm_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
echo "Building RPM for ${RPM_ARCH}"
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
sudo gem install fpm
./scripts/test_package_versions.sh --require-package-managers
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
depends_args=()
if [[ -n "$RPM_DEPENDS" ]]; then
depends_args+=(--depends "$RPM_DEPENDS")
fi
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://rustfs.com/docs/ for more information
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ROOT_USER=""
# RUSTFS_ROOT_PASSWORD=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
@@ -434,40 +462,18 @@ jobs:
--iteration "$RPM_RELEASE" \
--architecture "$RPM_ARCH" \
--package "$RPM_FILE" \
--depends "glibc >= 2.31" \
"${depends_args[@]}" \
--description "High-performance distributed object storage (${LIBC} build)" \
--maintainer "RustFS Team <support@rustfs.com>" \
--description "High-performance distributed object storage" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--after-install <(cat <<'POSTINST'
#!/bin/bash
set -e
if ! getent passwd rustfs > /dev/null 2>&1; then
useradd -r -s /bin/false -d /opt/rustfs rustfs
fi
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTINST
) \
--before-remove <(cat <<'PRERM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
systemctl stop rustfs
fi
PRERM
) \
--after-remove <(cat <<'POSTRM'
#!/bin/bash
set -e
if [ -d /run/systemd/system ]; then
systemctl daemon-reload
fi
POSTRM
) \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
@@ -489,9 +495,9 @@ jobs:
echo "✅ RPM built: $RPM_FILE"
- name: Upload packages to artifacts
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.arch }}
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
path: |
*.deb
*.rpm
@@ -587,10 +593,11 @@ jobs:
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# GitHub stores release asset names with '~' normalized to '.'
# (e.g. rustfs_1.0.0~rc.2_amd64.deb is stored as
# rustfs_1.0.0.rc.2_amd64.deb), so checksum entries must
# reference the name as stored on the release.
# GitHub stores release asset names with '~' normalized to '.'.
# Current package file names embed the raw tag and contain no
# '~', but older releases may still ship such assets, so the
# checksum entries must reference the name as stored on the
# release.
github_base="${base//\~/.}"
# Remove any stale entry (both naming variants), then append
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
@@ -612,7 +619,7 @@ jobs:
name: Summary
needs: [ resolve, package ]
if: always()
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 5
steps:
- name: Print summary
+4 -4
View File
@@ -57,7 +57,7 @@ jobs:
timeout-minutes: 180
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0 # baseline may be an earlier successful scheduled head
@@ -94,7 +94,7 @@ jobs:
- name: Find last successful scheduled baseline
id: scheduled_baseline
if: github.event_name == 'schedule'
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
result-encoding: string
script: |
@@ -271,7 +271,7 @@ jobs:
- name: Upload A/B results
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: hotpath-warp-ab-${{ github.run_number }}
# Includes per-cell median_summary.csv / baseline_compare.csv, both gates,
@@ -372,7 +372,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+3 -3
View File
@@ -43,11 +43,11 @@ concurrency:
jobs:
check-ephemerality:
name: Check runner ephemerality
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -72,7 +72,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
@@ -0,0 +1,221 @@
# RustFS Fault-Tolerance (degradation) Matrix Sweep
#
# Exhaustive topology x EC outage sweep, complementary to the scenario suite
# (rustfs-fault-tolerance-test.yml). For each topology (nodes x drives per
# node) and each EC level P in 1..N*drives/2 the runner installs that exact
# cluster shape and drives progressive node outages plus recovery:
#
# M-<topo>EC<P>-full full-capacity reads/writes must succeed
# M-<topo>EC<P>-n<k> with k nodes online: reads succeed iff the online
# drives cover the read quorum (N-P), writes iff they
# cover the write quorum (N-P+1); probes below quorum
# assert the expected outage
# M-<topo>EC<P>-rec after restoring every node, reads/writes succeed
#
# 9 topologies x EC levels = 36 combinations, 300 cases. The full sweep runs
# 3-5 hours and is dispatched on demand (e.g. after a nightly chain goes
# green), not nightly. Case ids and expectations come from product source
# (default_parity_count, erasure set sizing) via rustfs/auto-testing.
name: RustFS Fault-Tolerance Matrix
on:
workflow_dispatch:
inputs:
package_url:
description: 'Direct .deb URL. Required unless the nightly default is wanted.'
required: false
type: string
auto_testing_ref:
description: 'auto-testing branch, tag, or SHA (defaults to main)'
required: false
default: main
type: string
matrix_topo:
description: 'Topology filter like 4x2; empty sweeps all nine topologies'
required: false
default: ''
type: string
matrix_ec:
description: 'EC/parity filter; empty sweeps every EC level'
required: false
default: ''
type: string
cleanup_before:
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
type: boolean
default: true
cleanup_after:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
permissions:
contents: read
# The suite stops services and revokes drive access on the shared fleet; only one
# functional suite may touch the environment at a time.
concurrency:
group: rustfs-shared-functional-tests-v2
cancel-in-progress: false
defaults:
run:
shell: bash
env:
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES }}
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER }}
PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
jobs:
fault-tolerance-matrix:
runs-on: smoke-testing
# The full 38-combination sweep takes 3-5h; 360 is the runner job ceiling.
timeout-minutes: 360
steps:
- name: Initialize functional evidence
id: evidence
run: |
set -euo pipefail
umask 077
FUNCTIONAL_ARTIFACTS_DIR="${RUNNER_TEMP}/rustfs-fault-tolerance-matrix-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
mkdir -- "${FUNCTIONAL_ARTIFACTS_DIR}" "${FUNCTIONAL_ARTIFACTS_DIR}-scratch" "${FUNCTIONAL_ARTIFACTS_DIR}/evidence"
{
printf 'FUNCTIONAL_ARTIFACTS_DIR=%s\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'LOG_FILE=%s/suite.log\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'REPORT_FILE=%s/report.md\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'RESULTS_FILE=%s/results.json\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'EVIDENCE_DIR=%s/evidence\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
} >> "${GITHUB_ENV}"
- name: Checkout auto-testing scripts
timeout-minutes: 5
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ inputs.auto_testing_ref }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
path: auto-testing
persist-credentials: false
- name: Record auto-testing revision
run: |
set -euo pipefail
AUTO_TESTING_SHA="$(git -C auto-testing rev-parse HEAD)"
printf 'AUTO_TESTING_SHA=%s\n' "${AUTO_TESTING_SHA}" >> "${GITHUB_ENV}"
echo "auto-testing revision: ${AUTO_TESTING_SHA}"
- name: Show environment
run: |
uname -a
jq --version
aws --version
echo "auto-testing: ${AUTO_TESTING_SHA}"
df -h /data | tail -1
- name: Cleanup environment (before)
timeout-minutes: 5
if: ${{ inputs.cleanup_before != 'false' }}
run: |
./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}"
- name: Run matrix sweep (topology x EC outage combinations)
timeout-minutes: 350
id: test
# No continue-on-error: an UNEXPECTED verdict (a probe that should
# fail by quorum math but passes, or vice versa) is the finding this
# sweep exists to surface, and the red run is the signal until matrix
# runs get backlog integration like the scenario suite has.
env:
MATRIX_EC: ${{ inputs.matrix_ec }}
run: |
set -euo pipefail
ARGS=(--matrix -y --package-url "${{ inputs.package_url || env.RUSTFS_NIGHTLY_PACKAGE_URL }}" --log-file "${LOG_FILE}" --results-file "${RESULTS_FILE}")
if [ -n "${{ inputs.matrix_topo }}" ]; then
# Trailing value form: consumed as `--matrix-topo <topo> [ec]`.
ARGS+=(--matrix-topo "${{ inputs.matrix_topo }}")
if [ -n "${MATRIX_EC}" ]; then
ARGS+=("${MATRIX_EC}")
fi
fi
./auto-testing/rustfs-fault-tolerance-test.sh "${ARGS[@]}"
- name: Generate report
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
run: |
set -euo pipefail
{
echo "# RustFS fault-tolerance matrix report"
echo ""
echo "- Run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Package: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
echo "- auto-testing: \`${AUTO_TESTING_SHA}\`"
echo "- Topology filter: ${{ inputs.matrix_topo || 'all (10 topologies)' }}"
echo "- EC filter: ${{ inputs.matrix_ec || 'all levels' }}"
echo ""
echo "## Per-probe results"
echo ""
echo '```'
grep -E '^FT-(CASE|COMPLETE|SUMMARY|REPORT)' "${LOG_FILE}" || echo "(no FT-CASE lines found)"
echo '```'
echo ""
echo "## Health snapshots"
echo ""
for f in "${FUNCTIONAL_ARTIFACTS_DIR}"/evidence/*.code; do
[ -e "${f}" ] || continue
printf '%s -> %s\n' "$(basename "${f}" .code)" "$(cat "${f}")"
done
} > "${REPORT_FILE}"
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
# An unfiltered sweep must land exactly 300 cases (36 topology x EC
# combinations at 2*(nodes+1) cases each); filtered sweeps are held
# to internal consistency. Missing/harness-error rows always fail.
EXPECTED_TOTAL=""
if [ -z "${{ inputs.matrix_topo }}" ] && [ -z "${MATRIX_EC}" ]; then
EXPECTED_TOTAL=300
fi
jq -e --arg testing_sha "${AUTO_TESTING_SHA}" --argjson expected_total "${EXPECTED_TOTAL:-null}" '
.schema_version == 1 and
.auto_testing_sha == $testing_sha and
.complete == true and
.counts.expected == ($expected_total // .counts.expected) and
.counts.expected == .counts.seen and
.counts.harness_errors == 0 and
(.expected_cases | length) == .counts.expected and
(.expected_cases | unique | length) == .counts.expected and
(.seen_cases | length) == .counts.seen and
(.missing_cases | length) == 0 and
(.harness_errors | length) == 0 and
(.cases | length) == .counts.seen and
([.cases[].case_id] | unique | length) == .counts.seen and
([.cases[].case_id] | sort) == (.expected_cases | sort) and
all(.cases[]; (.case_id | startswith("M-")) and (.verdict == "pass" or .verdict == "known-divergence" or .verdict == "UNEXPECTED")) and
.counts.unexpected == ([.cases[] | select(.verdict == "UNEXPECTED")] | length) and
.counts.known_divergence == ([.cases[] | select(.verdict == "known-divergence")] | length)
' "${RESULTS_FILE}" >/dev/null
- name: Upload test logs & evidence
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-fault-tolerance-matrix-${{ github.run_id }}-${{ github.run_attempt }}
path: |
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/report.md
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/results.json
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/suite.log
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/evidence/
if-no-files-found: warn
- name: Cleanup environment (after)
timeout-minutes: 5
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}" || true
@@ -83,7 +83,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -113,7 +113,7 @@ jobs:
- name: Checkout auto-testing scripts
timeout-minutes: 5
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || inputs.auto_testing_ref || 'main' }}
@@ -149,7 +149,7 @@ jobs:
CHAIN_MANIFEST: ${{ inputs.chain_manifest }}
run: python3 scripts/prepare_functional_package.py prepare
- name: Run fault-tolerance scenarios (A, B, C, C2, D)
- name: Run fault-tolerance scenarios (A, B, C, C2, D, E)
timeout-minutes: 45
id: test
# Standalone case failures are reported by the backlog manager.
@@ -194,21 +194,21 @@ jobs:
jq -e --arg testing_sha "${AUTO_TESTING_SHA}" '
.schema_version == 1 and
.auto_testing_sha == $testing_sha and
.selected_scenarios == ["A", "B", "C", "C2", "D"] and
.selected_scenarios == ["A", "B", "C", "C2", "D", "E"] and
.complete == true and
.counts.expected == 38 and
.counts.seen == 38 and
.counts.expected == 44 and
.counts.seen == 44 and
.counts.harness_errors == 0 and
(.expected_cases | length) == 38 and
(.expected_cases | unique | length) == 38 and
(.seen_cases | length) == 38 and
(.seen_cases | unique | length) == 38 and
(.expected_cases | length) == 44 and
(.expected_cases | unique | length) == 44 and
(.seen_cases | length) == 44 and
(.seen_cases | unique | length) == 44 and
(.seen_cases | sort) == (.expected_cases | sort) and
(.missing_cases | length) == 0 and
(.harness_errors | length) == 0 and
(.cases | length) == 38 and
(.cases | length) == 44 and
([.cases[].case_id] | sort) == (.expected_cases | sort) and
([.cases[].case_id] | unique | length) == 38 and
([.cases[].case_id] | unique | length) == 44 and
all(.cases[]; .verdict == "pass" or .verdict == "known-divergence" or .verdict == "UNEXPECTED") and
.counts.unexpected == ([.cases[] | select(.verdict == "UNEXPECTED")] | length) and
.counts.known_divergence == ([.cases[] | select(.verdict == "known-divergence")] | length)
@@ -294,7 +294,7 @@ jobs:
- name: Upload test logs & evidence
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -386,7 +386,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}/fault-tolerance.json
+12 -9
View File
@@ -42,25 +42,28 @@ concurrency:
jobs:
prepare:
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.event == 'schedule' }}
runs-on: ubuntu-latest
runs-on: smoke-testing
timeout-minutes: 10
outputs:
manifest: ${{ steps.candidate.outputs.manifest }}
performance_ready: ${{ steps.perf_probe.outputs.performance_ready }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Resolve published candidate
id: candidate
env:
GH_TOKEN: ${{ github.token }}
# The staleness probe in resolve_functional_candidate.py reads the
# private rustfs/auto-testing repo; github.token cannot, and a probe
# failure silently keeps a stale testing_sha pinned for the chain.
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
BUILD_RUN_ATTEMPT: ${{ inputs.build_run_attempt }}
CHAIN_OUTPUT: ${{ runner.temp }}/chain-candidate.json
run: python3 scripts/resolve_functional_candidate.py
- name: Retain candidate identity
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-candidate-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-candidate.json
@@ -185,16 +188,16 @@ jobs:
complete-chain:
needs: [prepare, upgrade, s3, kms, tier, storage, heal, pool, security, replication, fault-tolerance, table, performance]
if: ${{ always() && needs.prepare.result != 'skipped' }}
runs-on: ubuntu-latest
runs-on: smoke-testing
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download suite evidence
id: download
continue-on-error: true
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: functional-chain-*-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-evidence
@@ -210,7 +213,7 @@ jobs:
--output "${RUNNER_TEMP}/chain-report.json"
- name: Retain chain report regardless of test verdict
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-report-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -228,7 +231,7 @@ jobs:
--directory "${RUNNER_TEMP}/chain-evidence"
--output "${RUNNER_TEMP}/chain-complete.json"
- name: Upload complete-chain evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-complete-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-complete.json
+4 -4
View File
@@ -66,7 +66,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -93,7 +93,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -366,7 +366,7 @@ jobs:
- name: Upload test logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-heal-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -467,7 +467,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-heal-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-heal-${{ github.run_id }}-${{ github.run_attempt }}/heal.json
+4 -4
View File
@@ -59,7 +59,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -85,7 +85,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -315,7 +315,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-kms-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -414,7 +414,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-kms-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-kms-${{ github.run_id }}-${{ github.run_attempt }}/kms.json
@@ -90,10 +90,10 @@ env:
jobs:
runner-check:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 5
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Check performance runner before entering its queue
@@ -116,7 +116,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -144,7 +144,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -307,7 +307,7 @@ jobs:
- name: Upload test logs & results
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-perf-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -349,7 +349,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-performance-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-performance-${{ github.run_id }}-${{ github.run_attempt }}/performance.json
+40 -6
View File
@@ -7,6 +7,11 @@ on:
description: Verified candidate and chain attempt from the chain driver
type: string
required: true
pool_timeout_minutes:
description: Wall-clock budget (minutes) for the pool run step
type: string
required: false
default: '240'
workflow_dispatch:
inputs:
rustfs_version:
@@ -39,6 +44,10 @@ on:
description: 'Run the pool decommission step (3-pool topology only)'
type: boolean
default: true
pool_timeout_minutes:
description: 'Wall-clock budget (minutes) for the whole pool run: fill + rebalance wait + decommission'
required: false
default: '240'
cleanup_before:
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
type: boolean
@@ -82,7 +91,8 @@ jobs:
pool-expansion-test:
name: Pool expansion / decommission test
runs-on: smoke-testing
timeout-minutes: 60
# Leave setup and finalizer headroom beyond the default 240-minute test.
timeout-minutes: 360
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
env:
RUSTFS_POOL_ADMIN_ENDPOINT: ${{ secrets.RUSTFS_POOL_ADMIN_ENDPOINT || vars.RUSTFS_POOL_ADMIN_ENDPOINT || 'http://rustfs-node1:9000' }}
@@ -92,7 +102,7 @@ jobs:
RUSTFS_POOL_NODE_ENDPOINTS: ${{ secrets.RUSTFS_POOL_NODE_ENDPOINTS || vars.RUSTFS_POOL_NODE_ENDPOINTS || 'http://rustfs-node1:9000 http://rustfs-node2:9000 http://rustfs-node3:9000 http://rustfs-node4:9000' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -104,7 +114,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -251,7 +261,12 @@ jobs:
- name: Run pool expansion & decommission test
id: pool_test
timeout-minutes: 45
# The suite budgets up to 24h each for rebalance and decommission
# (REBALANCE_TIMEOUT / DECOMMISSION_TIMEOUT in rustfs_pool_expand.sh);
# the previous 45m step cap killed four consecutive runs mid-rebalance
# with steps 1-7 green. 240m covers the observed pace (rebalance 1/4
# in ~21m) plus one decommission pass; override per dispatch.
timeout-minutes: ${{ inputs.pool_timeout_minutes || 240 }}
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
@@ -568,6 +583,25 @@ jobs:
echo "pool test script reported a failure (step FAIL verdict or die() abort)" >&2
failed=1
fi
# A runner-level kill (step timeout) truncates the suite without a
# FAIL verdict or an [ERROR] line, so require every expected
# [POOL-STEP] marker to be present with PASS - missing markers mean
# the run was truncated, not green. This also closes the gap for
# standalone dispatches, where the chain driver's evidence
# validation does not run.
last_step=6
if [ "${{ inputs.pools || '3' }}" = "3" ]; then
last_step=8
if [ "${{ inputs.run_decommission != 'false' }}" = "true" ]; then
last_step=9
fi
fi
for n in $(seq 1 "${last_step}"); do
grep -q "^\[POOL-STEP\] ${n} .* PASS" "${POOL_ARTIFACT_DIR}/pool-test.log" 2>/dev/null || {
echo "pool test log is missing [POOL-STEP] ${n} PASS (expected through step ${last_step}); the run was truncated" >&2
failed=1
}
done
[ "${failed}" -eq 0 ] || exit 1
- name: Upload functional report to dashboard
@@ -656,7 +690,7 @@ jobs:
- name: Upload test logs
timeout-minutes: 2
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-pool-test-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/rustfs-pool-${{ github.run_id }}-${{ github.run_attempt }}
@@ -764,7 +798,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-pool-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-pool-${{ github.run_id }}-${{ github.run_attempt }}/pool.json
@@ -71,7 +71,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -97,7 +97,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -323,7 +323,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-replication-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -419,7 +419,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-replication-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-replication-${{ github.run_id }}-${{ github.run_attempt }}/replication.json
+4 -4
View File
@@ -47,7 +47,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -73,7 +73,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -292,7 +292,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-s3-compat-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -391,7 +391,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-s3-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-s3-${{ github.run_id }}-${{ github.run_attempt }}/s3.json
+6 -6
View File
@@ -87,7 +87,7 @@ jobs:
# the workspace root would wipe the auto-testing clone above (that is
# exactly how run 33934141181 lost rustfs-security-test.sh).
- name: Checkout repository (for the OIDC live gate script)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
path: rustfs-repo
@@ -106,10 +106,10 @@ jobs:
if: ${{ inputs.chain_manifest != '' }}
env:
CHAIN_MANIFEST: ${{ inputs.chain_manifest }}
run: python3 scripts/functional_chain_evidence.py consume
run: python3 rustfs-repo/scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -325,7 +325,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-security-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -389,13 +389,13 @@ jobs:
CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }}
CHAIN_REPORT_OUTCOME: ${{ steps.report.outcome }}
run: >-
python3 scripts/functional_chain_evidence.py record --suite security
python3 rustfs-repo/scripts/functional_chain_evidence.py record --suite security
--report "${SECURITY_ARTIFACTS_DIR}/suite-report.md"
--output "${RUNNER_TEMP}/chain-security-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/security.json"
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-security-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-security-${{ github.run_id }}-${{ github.run_attempt }}/security.json
+4 -4
View File
@@ -56,7 +56,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -82,7 +82,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -307,7 +307,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-storage-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -406,7 +406,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-storage-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-storage-${{ github.run_id }}-${{ github.run_attempt }}/storage.json
+5 -5
View File
@@ -46,7 +46,7 @@ jobs:
timeout-minutes: 60
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -73,7 +73,7 @@ jobs:
- name: Checkout auto-testing scripts
timeout-minutes: 5
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -133,7 +133,7 @@ jobs:
if [ -n "${PACKAGE_URL}" ]; then
ARGS+=(--package-url "${PACKAGE_URL}")
elif [ -n "${RUSTFS_VERSION}" ] && [ "${RUSTFS_VERSION}" != "null" ]; then
ARGS+=(--package-url "https://github.com/rustfs/rustfs/releases/download/${RUSTFS_VERSION}/rustfs_${RUSTFS_VERSION}_amd64.deb")
ARGS+=(--package-url "https://github.com/rustfs/rustfs/releases/download/${RUSTFS_VERSION}/rustfs-linux-x86_64-gnu-v${RUSTFS_VERSION}.deb")
else
ARGS+=(--package-url "${RUSTFS_NIGHTLY_PACKAGE_URL}")
fi
@@ -281,7 +281,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-table-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -353,7 +353,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-table-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-table-${{ github.run_id }}-${{ github.run_attempt }}/table.json
+4 -4
View File
@@ -71,7 +71,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -95,7 +95,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -460,7 +460,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-tier-test-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ env.TIER_ARTIFACTS_DIR }}/
@@ -734,7 +734,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-tier-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-tier-${{ github.run_id }}-${{ github.run_attempt }}/tier.json
+5 -5
View File
@@ -89,7 +89,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -115,7 +115,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -207,7 +207,7 @@ jobs:
local version="$1" tag asset url
[ -n "${version}" ] && [ "${version}" != "null" ] || return 0
tag="${version#v}"
asset="rustfs_${tag//-/.}_amd64.deb"
asset="rustfs-linux-x86_64-gnu-v${tag}.deb"
url="https://github.com/rustfs/rustfs/releases/download/${tag}/${asset}"
if ! gh api "repos/rustfs/rustfs/releases/tags/${tag}" --jq '.assets[].name' 2>/dev/null | grep -qxF "${asset}"; then
echo "ERROR: release ${tag} has no downloadable asset ${asset}:" >&2
@@ -400,7 +400,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-upgrade-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -503,7 +503,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-upgrade-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-upgrade-${{ github.run_id }}-${{ github.run_attempt }}/upgrade.json
@@ -44,7 +44,7 @@ permissions:
jobs:
forced-failure:
name: Forced failure
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 5
steps:
- name: Fail on purpose
@@ -64,7 +64,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
@@ -29,6 +29,9 @@ concurrency:
jobs:
check-freshness:
name: Check scheduled validation freshness
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
# scheduled validations it watches stall.
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
@@ -36,7 +39,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Check latest scheduled runs
@@ -42,6 +42,9 @@ jobs:
github.event.workflow_run.event == 'schedule' &&
github.event.workflow_run.conclusion != 'success' &&
github.event.workflow_run.conclusion != 'failure'
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
# scheduled validations it watches stall.
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
@@ -49,7 +52,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update incomplete-run issue
+2 -2
View File
@@ -26,10 +26,10 @@ on:
jobs:
stale:
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 30
steps:
- uses: actions/stale@5bef64f19d7facfb25b37b414482c7164d639639 # v9
- uses: actions/stale@4391f3da665fdf50b6810c1a66712fb9ba21aa93 # v11.0.0
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
stale-issue-message: 'This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.'
+1 -1
View File
@@ -14,7 +14,7 @@ concurrency:
jobs:
update:
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 10
steps:
- uses: overtrue/repo-visuals-action@ee2c632f6ce617e851fb46ea935ee8af762ebb93 # v1.4.0
+4 -4
View File
@@ -41,7 +41,7 @@ env:
jobs:
targets-live:
name: PostgreSQL, MySQL, AMQP, and NATS
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
@@ -52,7 +52,7 @@ jobs:
RUSTFS_TEST_NATS_URL: nats://127.0.0.1:4222
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -166,7 +166,7 @@ jobs:
- name: Upload target integration diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: targets-integration-${{ github.run_number }}-${{ github.run_attempt }}
path: artifacts/targets-live
@@ -183,7 +183,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+1 -1
View File
@@ -56,7 +56,7 @@ jobs:
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
+1
View File
@@ -25,6 +25,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Presigned URLs honour only signed headers** (GHSA-g8w9-qw9q-fghr): a SigV4 presigned request that carries an `x-amz-*` request header not listed in `X-Amz-SignedHeaders` is now rejected with `403 AccessDenied` ("There were headers present in the request which were not signed"), matching AWS S3. Previously the holder of a presigned `PutObject` URL could add unsigned `x-amz-tagging`, `x-amz-storage-class`, `x-amz-website-redirect-location`, ACL, metadata, Object Lock or SSE headers and have them applied. Presigners that intend a property must set it before signing so the SDK lists the header in `SignedHeaders`; `x-amz-cf-id` (CloudFront) remains tolerated unsigned. Header-signed SigV4 and SigV2 requests are unchanged.
### Fixed
- **Two admin handlers reported a missing policy or user as `500 InternalError`**: `GET /rustfs/admin/v3/info-canned-policy` and `DELETE /rustfs/admin/v3/remove-user` wrapped every IAM error as an internal fault instead of classifying it, so a lookup for something that does not exist answered `500 InternalError` where the rest of the admin API answers `404 NoSuchResource` — `GET /rustfs/admin/v3/user-info` already did, and `iam_error_to_s3_error` already mapped `NoSuchPolicy`, `NoSuchUser`, `NoSuchServiceAccount` and `NoSuchTempAccount` to `NoSuchResource`. A client could not tell "does not exist" from a real server failure, and a controller that checks existence before creating (a Kubernetes operator reconciling policies or users) could never create the object: the missing-policy lookup failed with a `500` and it retried forever. Both handlers now map their IAM errors through the shared `iam_error_to_s3_error` helper, so a missing policy or user is `404 NoSuchResource` and the `error` is preserved as the S3 error's source; the three IAM calls in the removal path (temporary-user lookup, service-account lookup, delete) are all reclassified, and every other IAM error stays `500 InternalError`. Fixes #8117.
- **Vault backends accepted key names that escape the key prefix**: only the Local backend refused a key identifier containing a path separator, so on Vault KV2 `POST /rustfs/admin/v3/kms/keys` with a name such as `bad/name` succeeded and created a nested KV2 path that the key listing then reported as a directory rather than a key, and an identifier containing `..` addressed a record outside the configured key prefix once the HTTP client normalised the URL. Both Vault backends now apply the Local backend's containment rule at the point where the identifier becomes a path or a Transit key name: an empty identifier, one containing `/`, `\\` or NUL, or the dot segments `.` and `..` is refused with `400` (`InvalidKey`) before any request reaches Vault. The AWS backend is unaffected, since it addresses keys by ARN and alias. A Vault deployment that created such a key on an earlier build must re-create it under a plain name; objects encrypted under the old name stay readable only until that key is refused, so migrate them first. Refs rustfs/backlog#2474.
- **Fresh multi-pool bootstrap with distinct format creators**: a new deployment whose pools have their first endpoint on different nodes (for example two single-node pools) could never publish its initial `pool.bin`: each node held fresh-bootstrap proof only for the pool it formatted, the deployment-wide proof collapsed to none, and every node died with `pool metadata recovery required: no durable bootstrap identity or pool.bin replica is available` after the startup retry budget. The first pool's creator now mints the pending cluster identity on its own pool, every other creator copies that nonce-bound identity onto the pool it formatted first-hand, and the elected writer publishes `pool.bin` once every pool replica carries the same pending identity. Corrupt or disagreeing replicas, pools that merely have a format, expansion pools joining an initialized deployment, and restarts without first-hand proof still fail closed. Non-elected nodes that start before `pool.bin` exists, and the elected writer while it waits for the other creators, no longer latch their pool-metadata write gate for the life of the process. Refs rustfs/backlog#2338, rustfs/backlog#2375.
- **Lock RPC timeout storms** (#7363): the remote lock client no longer evicts and re-dials the shared internode HTTP/2 channel on every request deadline. A timeout evicts only when the peer has not completed any lock RPC for two deadlines, evictions and transport-failure re-dials are rate limited per peer (`RUSTFS_OBJECT_LOCK_RPC_EVICTION_COOLDOWN_MS`, default 5 s), and a timed-out request is left running instead of being reset (bounded per peer by `RUSTFS_OBJECT_LOCK_RPC_DETACHED_LIMIT`, default 256), so a slow lock endpoint can no longer drive the `RST_STREAM`/`GOAWAY too_many_resets`/reconnect loop. A lock granted after its caller timed out is released immediately, and unlocks that fail the quick retries continue on a deferred 1/2/4/8/16 s schedule before the server lease reclaims them. New `rustfs_remote_lock_*` metrics cover timeouts, evictions, suppressed evictions, detached streams, late completions and late releases per peer. Operator guide at `docs/operations/lock-rpc-storm-protection.md`.
Generated
+446 -398
View File
File diff suppressed because it is too large Load Diff
+35 -33
View File
@@ -20,6 +20,7 @@ members = [
"crates/common", # Shared utilities and data structures
"crates/data-usage", # Shared data usage models and algorithms
"crates/config", # Configuration management
"crates/connect-symbol-catalog", # Release-only CPU profile symbol catalog generator
"crates/credentials", # Credential management system
"crates/crypto", # Cryptography and security features
"crates/ecstore", # Erasure coding storage implementation
@@ -156,10 +157,10 @@ futures-lite = "2.6.1"
futures-util = "0.3.34"
pollster = "1.0.1"
pulsar = { default-features = false, version = "6.9.0" }
lapin = { default-features = false, version = "4.11.0" }
lapin = { default-features = false, version = "4.12.0" }
hyper = { version = "1.11.1" }
hyper-rustls = { default-features = false, version = "0.27.9" }
hyper-util = { version = "0.1.20" }
hyper-rustls = { default-features = false, version = "0.27.10" }
hyper-util = { version = "0.1.21" }
http = "1.5.0"
http-body = "1.1.0"
http-body-util = "0.1.5"
@@ -171,7 +172,7 @@ socket2 = { version = "0.6.5" }
tokio = { version = "1.53.1" }
tokio-rustls = { default-features = false, version = "0.26.5" }
tokio-stream = { version = "0.1.19" }
tokio-test = "0.4.5"
tokio-test = "0.4.6"
tokio-util = { version = "0.7.19" }
tonic = { version = "0.14.6" }
tonic-prost = { version = "0.14.6" }
@@ -244,13 +245,13 @@ atoi = "3.1.0"
atomic_enum = "0.3.0"
aws-config = { version = "1.12.0" }
aws-credential-types = { version = "1.3.0" }
aws-sdk-kms = { default-features = false, version = "1.120.0" }
aws-sdk-s3 = { default-features = false, version = "1.148.0" }
aws-sdk-sts = { default-features = false, version = "1.116.0" }
aws-sdk-kms = { default-features = false, version = "1.122.0" }
aws-sdk-s3 = { default-features = false, version = "1.150.0" }
aws-sdk-sts = { default-features = false, version = "1.118.0" }
aws-smithy-async = { version = "1.3.0" }
aws-smithy-http-client = { default-features = false, version = "1.4.2" }
aws-smithy-runtime-api = { version = "1.16.2" }
aws-smithy-types = { version = "1.6.4" }
aws-smithy-runtime-api = { version = "1.18.0" }
aws-smithy-types = { version = "1.8.1" }
base64-simd = "0.8.0"
brotli = "9.0.0"
clap = { version = "4.6.7" }
@@ -265,11 +266,11 @@ datafusion = { default-features = false, version = "55.1.0" }
derive_builder = "0.20.2"
ed25519-dalek = "3.0.0"
enumset = "1.1.14"
faster-hex = "0.10.0"
faster-hex = "1.0.0"
flate2 = "1.1.10"
glob = "0.3.4"
google-cloud-storage = "1.18.0"
google-cloud-auth = "1.16.0"
google-cloud-storage = "1.19.0"
google-cloud-auth = "1.17.0"
hashbrown = { version = "0.17.1" }
# Base32 for RFC 6238 TOTP shared secrets (RFC 4648 unpadded, the alphabet
# every authenticator app expects). Already in the graph transitively.
@@ -292,6 +293,7 @@ moka = { version = "0.12.16" }
netif = "0.1.6"
num_cpus = { version = "1.17.0" }
nvml-wrapper = "0.13.0"
object = { version = "=0.40.0", default-features = false }
parking_lot = "0.12.5"
path-absolutize = "4.0.1"
percent-encoding = "2.3.2"
@@ -301,24 +303,24 @@ percent-encoding = "2.3.2"
qrcode-rs = { version = "2.1.1", default-features = false, features = ["std", "svg"] }
pin-project-lite = "0.2.17"
pretty_assertions = "1.4.1"
rand = { version = "0.10.2" }
rand = { version = "0.10.3" }
ratelimit = "2.0.0"
rayon = "1.12.0"
rustfs-erasure-codec = { version = "9.0.0" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.34.0" }
redis = { version = "1.7.0" }
redis = { version = "1.7.1" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.5" }
rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
s3s = { version = "0.16.1", features = ["minio"] }
s3s-sigv4 = { version = "0.16.1" }
s3s = { version = "0.17.0", features = ["minio"] }
s3s-sigv4 = "0.17.0"
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.3"
smallvec = { version = "1.16.1" }
siphasher = "1.0.4"
smallvec = { version = "1.16.2" }
compact_str = "0.10.0"
snap = "1.1.2"
starshard = { version = "2.3.0" }
@@ -326,13 +328,13 @@ strum = { version = "0.28.0" }
sysinfo = "0.39.6"
temp-env = "0.3.6"
tempfile = "3.27.0"
test-case = "3.3.1"
thiserror = "2.0.20"
test-case = "3.4.0"
thiserror = "2.0.21"
tracing = { version = "0.1.44" }
tracing-appender = "0.2.5"
tracing-core = "0.1.36"
tracing-error = "0.2.1"
tracing-opentelemetry = { version = "0.33" }
tracing-opentelemetry = { version = "0.34" }
tracing-subscriber = { version = "0.3.23" }
transform-stream = "0.3.1"
url = "2.5.8"
@@ -351,20 +353,20 @@ zstd = "0.14.0"
# Observability and Metrics
metrics = "0.24.6"
metrics-util = "0.20"
dial9 = { version = "0.5.0", default-features = false }
opentelemetry = { version = "0.32.0" }
opentelemetry-appender-tracing = { version = "0.32.0" }
opentelemetry-otlp = { version = "0.32.0" }
opentelemetry-proto = { version = "0.32.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
opentelemetry_sdk = { version = "0.32.1" }
opentelemetry-semantic-conventions = { version = "0.32.1" }
opentelemetry-stdout = { version = "0.32.0" }
dial9 = { version = "0.5.2", default-features = false }
opentelemetry = { version = "0.33.0" }
opentelemetry-appender-tracing = { version = "0.33.0" }
opentelemetry-otlp = { version = "0.33.0" }
opentelemetry-proto = { version = "0.33.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
opentelemetry_sdk = { version = "0.33.0" }
opentelemetry-semantic-conventions = { version = "0.33.0" }
opentelemetry-stdout = { version = "0.33.0" }
pyroscope = { version = "2.1.1" }
# FTP and SFTP
libunftp = { version = "0.23.0" }
libunftp = { version = "0.23.1" }
unftp-core = "0.1.0"
suppaftp = { version = "12.0.0" }
suppaftp = { version = "12.1.0" }
rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.63.3" }
russh-sftp = "3.0.0"
@@ -373,9 +375,9 @@ russh-sftp = "3.0.0"
dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.5.4" }
rustfs-mimalloc = { version = "0.5.6" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.26.0", default-features = false }
hotpath = { version = "0.26.1", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
+3 -1
View File
@@ -106,7 +106,9 @@ RUN chmod +x /usr/bin/rustfs /entrypoint.sh
RUN addgroup -g 10001 -S rustfs && \
adduser -u 10001 -G rustfs -S rustfs -D && \
mkdir -p /data /logs && \
mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs && \
ln -s /data/.rustfs/events /opt/rustfs/events && \
ln -s /data/.rustfs/audit /opt/rustfs/audit && \
chown -R rustfs:rustfs /data /logs && \
chmod 0750 /data /logs
+3 -1
View File
@@ -44,7 +44,9 @@ RUN set -eux; \
WORKDIR /app
RUN set -eux; \
mkdir -p /data /logs; \
mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs; \
ln -s /data/.rustfs/events /opt/rustfs/events; \
ln -s /data/.rustfs/audit /opt/rustfs/audit; \
chown -R rustfs:rustfs /data /logs /app; \
chmod 0750 /data /logs
+3 -1
View File
@@ -110,7 +110,9 @@ RUN chmod +x /usr/bin/rustfs /entrypoint.sh
RUN groupadd -g 10001 rustfs && \
useradd -u 10001 -g rustfs -m -s /sbin/nologin rustfs && \
mkdir -p /data /logs && \
mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs && \
ln -s /data/.rustfs/events /opt/rustfs/events && \
ln -s /data/.rustfs/audit /opt/rustfs/audit && \
chown -R rustfs:rustfs /data /logs && \
chmod 0750 /data /logs
+3 -1
View File
@@ -240,7 +240,9 @@ WORKDIR /app
# Prepare data/log directories with sane defaults
RUN set -eux; \
mkdir -p /data /logs; \
mkdir -p /data/.rustfs/events /data/.rustfs/audit /logs /opt/rustfs; \
ln -s /data/.rustfs/events /opt/rustfs/events; \
ln -s /data/.rustfs/audit /opt/rustfs/audit; \
chown -R rustfs:rustfs /data /logs /app; \
chmod 0750 /data /logs
+1 -1
View File
@@ -186,7 +186,7 @@
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2024 Beijing Henghesha Technology Co., Ltd.
Copyright 2024-2026 RustFS, Inc.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
+2 -2
View File
@@ -141,7 +141,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -195,7 +195,7 @@ docker run -d --name rustfs -p 9000:9000 \
Notes:
- `RUSTFS_NOTIFY_ENABLE=true` enables the global notify module switch.
- For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
- If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
- If queue dir is omitted, the official image maps `/opt/rustfs/events` into the persistent `/data` volume so it is writable by the runtime user and pending events survive container recreation. Override `queue_dir` when using another deployment layout.
- `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
- Since `1.0.0-beta.11`, webhook endpoints on private or container networks
(`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are
+1 -1
View File
@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.1
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
+96
View File
@@ -163,6 +163,15 @@ pub struct MrfDurableRepairAnchor {
pub bucket_incarnation_id: Uuid,
}
/// A completed deep check proved that a legacy object is present and healthy,
/// but the format has no independent identity commitment that can discharge a
/// durable partial-write responsibility. Keep the journal entry and pause
/// automatic retries until an operator has repaired the source of truth.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MrfUnverifiedLegacyEvent {
pub anchor: MrfDurableRepairAnchor,
}
impl MrfDurableRepairAnchor {
/// Build a dischargeable anchor only when the caller supplies the storage
/// incarnation and the original ingress lease. Legacy replay records lack
@@ -782,6 +791,8 @@ const MRF_REPAIRED_EVENT_CAP: usize = 4096;
static MRF_REPAIRED_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfRepairedEvent>>> = OnceLock::new();
static MRF_VERIFIED_REPAIR_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfVerifiedRepairEvent>>> =
OnceLock::new();
static MRF_UNVERIFIED_LEGACY_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfUnverifiedLegacyEvent>>> =
OnceLock::new();
/// Record a legacy notification for compatibility. This is not an
/// acknowledgement of storage verification or durable repair completion.
@@ -859,6 +870,44 @@ pub fn take_mrf_verified_repair_events_for(bucket: &str) -> Vec<MrfVerifiedRepai
taken
}
/// Record a bounded, exact-identity notice that a legacy durable partial-write
/// was checked but cannot be proven from its on-disk format. This is not a
/// repair proof and must never release the durable responsibility.
pub fn note_mrf_unverified_legacy(event: MrfUnverifiedLegacyEvent) {
let registry = MRF_UNVERIFIED_LEGACY_EVENTS.get_or_init(|| std::sync::Mutex::new(std::collections::VecDeque::new()));
let Ok(mut events) = registry.lock() else {
return;
};
if events.contains(&event) {
return;
}
if events.len() >= MRF_REPAIRED_EVENT_CAP {
events.pop_front();
}
events.push_back(event);
}
/// Take held-legacy notices for one bucket. The durable ledger still verifies
/// the full anchor before changing the journal entry to its held state.
pub fn take_mrf_unverified_legacy_events_for(bucket: &str) -> Vec<MrfUnverifiedLegacyEvent> {
let Some(registry) = MRF_UNVERIFIED_LEGACY_EVENTS.get() else {
return Vec::new();
};
let Ok(mut events) = registry.lock() else {
return Vec::new();
};
let mut taken = Vec::new();
events.retain(|event| {
if event.anchor.bucket.as_ref() == bucket {
taken.push(event.clone());
false
} else {
true
}
});
taken
}
#[cfg(test)]
mod tests {
use super::*;
@@ -1233,6 +1282,53 @@ mod tests {
assert!(take_mrf_verified_repair_events_for("verified-bucket-a").is_empty());
assert_eq!(take_mrf_verified_repair_events_for("verified-bucket-b").len(), 1);
}
#[test]
fn unverified_legacy_events_are_bucket_scoped_and_bounded() {
let bucket = Arc::<str>::from("unverified-legacy-event-bucket");
let _ = take_mrf_unverified_legacy_events_for(bucket.as_ref());
let event = MrfUnverifiedLegacyEvent {
anchor: MrfDurableRepairAnchor {
kind: MrfKind::PartialWrite,
bucket: bucket.clone(),
object: Arc::from("unverified-object"),
version_id: None,
scope: Some(MrfScope {
pool_index: 1,
set_index: 2,
}),
delete_marker_purge: None,
lease: MrfIngressLease::new(912),
bucket_incarnation_id: Uuid::new_v4(),
},
};
note_mrf_unverified_legacy(event.clone());
note_mrf_unverified_legacy(event.clone());
assert_eq!(take_mrf_unverified_legacy_events_for(bucket.as_ref()), vec![event]);
assert!(take_mrf_unverified_legacy_events_for(bucket.as_ref()).is_empty());
let cap_bucket = Arc::<str>::from("unverified-legacy-event-cap-bucket");
for index in 0..MRF_REPAIRED_EVENT_CAP + 4 {
note_mrf_unverified_legacy(MrfUnverifiedLegacyEvent {
anchor: MrfDurableRepairAnchor {
kind: MrfKind::PartialWrite,
bucket: cap_bucket.clone(),
object: Arc::from(format!("object-{index}")),
version_id: None,
scope: Some(MrfScope {
pool_index: 1,
set_index: 2,
}),
delete_marker_purge: None,
lease: MrfIngressLease::new(u64::try_from(index).expect("test index fits")),
bucket_incarnation_id: Uuid::new_v4(),
},
});
}
let events = take_mrf_unverified_legacy_events_for(cap_bucket.as_ref());
assert_eq!(events.len(), MRF_REPAIRED_EVENT_CAP);
assert_eq!(events.first().expect("bounded ring is non-empty").anchor.object.as_ref(), "object-4");
}
}
#[tokio::test]
async fn partial_write_durable_admission_waits_for_persistence_and_renews_lease() {
+2 -2
View File
@@ -96,11 +96,11 @@ pub struct WorkloadAdmissionSnapshot {
pub class: WorkloadClass,
/// Current admission state.
pub state: AdmissionState,
/// Active work count when the owner exposes it.
/// Active work count or allocated/reserved permit units, as defined by the owner.
pub active: Option<usize>,
/// Queued work count when the owner exposes it.
pub queued: Option<usize>,
/// Admission limit when the owner exposes it.
/// Admission limit in the same units as `active`, when the owner exposes it.
pub limit: Option<usize>,
/// Optional state reason for disabled, throttled, saturated, or unknown states.
pub reason: Option<String>,
+11 -9
View File
@@ -169,15 +169,17 @@ Scanner cycle budget controls:
## Foreground write admission environment variables
Large direct `PutObject` requests and multipart `UploadPart` requests share one
per-process permit pool that bounds how many bodies are ingested and written
concurrently. Small direct PUTs stay on the legacy path.
per-process permit pool that bounds concurrent body ingest and storage writes.
Small direct PUTs stay on the legacy path.
- `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE`
- enables the default-on pool; `false` keeps only the soft request counter.
- default is `true`.
- `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT`
- permits in the pool; `0` derives half of `RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS`, clamped to `32`.
- default is `0` (32 permits at stock settings).
- a positive value is an exact request-count limit shared by all gated writes.
- default is `0`: derive half of `RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS`, clamped to `32`, as large-write slots. Each slot has eight units. A gated direct PUT or unknown-size multipart part uses eight units; known-size parts use one unit per 8 MiB, rounded up, with a minimum of one and maximum of eight.
- stock settings therefore share 256 units across at most 32 large/unknown writes or 256 parts of up to 8 MiB. Mixed sizes consume the same budget. Admitted known parts of up to 64 MiB total at most 2 GiB of declared payload; larger streaming writes and queued bodies are separate. This is not a bound on total process or kernel memory.
- admission snapshots report allocated or reserved units in `active` and the unit budget in `limit` under automatic sizing. Explicit and strict limits still report request-count permits.
- `RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_MIN_SIZE_BYTES`
- smallest direct `PutObject` that takes a permit; unknown-size requests always do.
- default is `33554432` (32 MiB).
@@ -193,8 +195,8 @@ concurrently. Small direct PUTs stay on the legacy path.
- RustFS does not read the request body while a part is queued, so the client's socket write stalls for the whole wait and whatever timeout the client or an intermediary has configured competes with this value. Keep it with margin below the shortest such timeout in use (botocore applies its 60 s `connect_timeout` to the body write; the AWS SDK for Java v2 has a 30 s socket write timeout; reverse proxies add their own body timeouts); a wait that outlives the client timeout surfaces as a dropped connection instead of `SlowDown`.
- `RUSTFS_PUT_MULTIPART_FOREGROUND_ADMISSION_MAX_PENDING`
- maximum `UploadPart` requests waiting for a permit at once; parts beyond it return `SlowDown` without waiting.
- default is `0`, which derives 16 times the permit limit (512 at stock settings).
- each queued HTTP/1 part holds whatever unread body the client already pushed into the connection's kernel receive buffer (an HTTP/2 part holds up to its flow-control window in process memory), so this depth also bounds that memory. RustFS leaves the receive buffer to kernel autotuning (see `RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES` below), which keeps an unread connection at the kernel's initial size (128 KiB on current Linux).
- default is `0`, which derives 16 times the large-write slot count, or the explicit request-count limit (512 at stock settings). Automatic subdivision does not enlarge this queue.
- each queued HTTP/1 part holds whatever unread body the client already pushed into the connection's kernel receive buffer (an HTTP/2 part holds up to its flow-control window in process memory). Autotuned receive buffers can remain large on reused connections; queue depth does not imply a fixed per-connection memory cost.
- `RUSTFS_PUT_FOREGROUND_ADMISSION_ENABLE`, `RUSTFS_PUT_FOREGROUND_ADMISSION_LIMIT`, `RUSTFS_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS`
- experimental strict gate that applies to every foreground write regardless of size and replaces the pool above when enabled.
- default is disabled; enabling it with limit `0` disables foreground write admission entirely.
@@ -203,9 +205,9 @@ concurrently. Small direct PUTs stay on the legacy path.
- `RUSTFS_HTTP_SOCKET_RECV_BUFFER_BYTES`
- fixed `SO_RCVBUF` for the API listener, inherited by every accepted socket; `0` leaves the receive buffer to kernel autotuning.
- default is `0`. Earlier releases hard-coded 4 MiB, which Linux doubles to 8 MiB and which disables autotuning, so every connection whose body was not being read yet (a multipart part queued for a foreground write permit) could accumulate up to 8 MiB of unread body in kernel memory; at SDK-default multipart concurrency that was enough to push a node into TCP memory pressure.
- with autotuning the per-connection receive ceiling is the kernel's (`net.ipv4.tcp_rmem` max, 6 MiB on stock Linux) instead of the former fixed 8 MiB, so a single very high-bandwidth-delay connection may see a somewhat lower ceiling; raise `net.ipv4.tcp_rmem` first, and set this variable only on kernels without receive-buffer autotuning (illumos/Solaris) or where the sysctl cannot be changed.
- the send buffer stays fixed at 4 MiB because the stock Linux send autotuning ceiling (`net.ipv4.tcp_wmem` max, 4 MiB) is lower than a GB-level response stream needs.
- default is `0`. A fixed buffer disables receive autotuning. Linux doubles the requested value for socket-memory accounting, subject to kernel limits; that capacity is not a measurement of queued payload or allocated memory.
- with autotuning the receive ceiling is controlled by the kernel (`net.ipv4.tcp_rmem` on Linux); its defaults vary with kernel version and memory. A connection can retain a buffer enlarged by previous requests while its next request is queued. Tune the kernel ceiling first, and set this variable only on kernels without receive-buffer autotuning (illumos/Solaris) or where the sysctl cannot be changed.
- the send buffer remains fixed at 4 MiB; this setting only controls the receive side.
## Remote tier timeout environment variables
+7 -4
View File
@@ -364,11 +364,14 @@ const _: () = assert!(!DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE);
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE";
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_ENABLE: bool = true;
/// Maximum automatic foreground write requests admitted concurrently per process.
/// Explicit maximum foreground write requests admitted concurrently per process.
///
/// `0` derives a conservative default from the local disk-read scheduler cap,
/// currently clamped to protect the commit path without making ordinary high
/// throughput uploads single-file.
/// `0` derives up to 32 large-write slots from the local disk-read scheduler.
/// Each automatic slot has eight units: a gated direct PUT or unknown-size
/// part uses all eight, while a known-size part uses one unit per 8 MiB,
/// rounded up and capped at eight. Thus stock settings share one budget across
/// at most 32 large/unknown writes or 256 parts of up to 8 MiB. A positive
/// override retains request-count semantics for every gated write.
pub const ENV_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: &str = "RUSTFS_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT";
pub const DEFAULT_PUT_LARGE_FOREGROUND_ADMISSION_LIMIT: usize = 0;
+4 -7
View File
@@ -164,13 +164,10 @@ pub const DEFAULT_HTTP1_MAX_BUF_SIZE: usize = 64 * 1024; // 64 KB
/// autotuning.
///
/// A fixed `SO_RCVBUF` is inherited by every accepted socket and disables
/// receive-buffer autotuning, so a connection whose request body is not being
/// read yet (a multipart part queued for a foreground write permit) lets up to
/// the fixed size of unread body accumulate in kernel memory — Linux doubles
/// the requested value, so the former hard-coded 4 MiB held up to 8 MiB per
/// queued connection (issue #7385). Autotuning keeps an unread connection at
/// the kernel's initial size and grows only connections that are being
/// drained. Set this only on kernels without receive-buffer autotuning
/// receive-buffer autotuning. Queued multipart bodies can accumulate in kernel
/// memory with either policy: a reused autotuned connection may already have
/// a large buffer. Socket capacity is not the same as queued payload or actual
/// memory allocation. Set this only on kernels without receive-buffer autotuning
/// (illumos/Solaris) or on very high-bandwidth-delay links where the kernel's
/// autotuning ceiling (`net.ipv4.tcp_rmem` on Linux) is too low and cannot be
/// raised.
+14
View File
@@ -30,6 +30,9 @@ pub const ENV_OBS_ENDPOINT_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_TIMEOUT_M
pub const ENV_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_TRACES_TIMEOUT_MILLIS";
pub const ENV_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_METRICS_TIMEOUT_MILLIS";
pub const ENV_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS: &str = "RUSTFS_OBS_ENDPOINT_LOGS_TIMEOUT_MILLIS";
/// Absolute path to a PEM bundle whose certificates are added to the system
/// trust roots used by OTLP/HTTP exporters.
pub const ENV_OBS_TLS_CA_FILE: &str = "RUSTFS_OBS_TLS_CA_FILE";
pub const ENV_OBS_PROFILING_ENDPOINT: &str = "RUSTFS_OBS_PROFILING_ENDPOINT";
pub const ENV_OBS_USE_STDOUT: &str = "RUSTFS_OBS_USE_STDOUT";
pub const ENV_OBS_SAMPLE_RATIO: &str = "RUSTFS_OBS_SAMPLE_RATIO";
@@ -55,6 +58,13 @@ pub const ENV_OBS_LOG_FILENAME: &str = "RUSTFS_OBS_LOG_FILENAME";
pub const ENV_OBS_LOG_ROTATION_TIME: &str = "RUSTFS_OBS_LOG_ROTATION_TIME";
pub const ENV_OBS_LOG_KEEP_FILES: &str = "RUSTFS_OBS_LOG_KEEP_FILES";
/// Span lifecycle events emitted by the JSON log layers (`none`|`close`|`full`).
///
/// Defaults to `none`: a record per span lifecycle multiplies log volume on
/// object hot paths, and emitting the ancestor span list on every record
/// produced multi-megabyte lines that journald truncated (backlog#2642).
pub const ENV_OBS_LOG_SPAN_EVENTS: &str = "RUSTFS_OBS_LOG_SPAN_EVENTS";
/// Log cleanup related configurations
pub const ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES";
pub const ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES";
@@ -99,6 +109,8 @@ pub const DEFAULT_OBS_LOG_DRY_RUN: bool = false;
pub const DEFAULT_OBS_LOG_MATCH_MODE_PREFIX: &str = "prefix";
pub const DEFAULT_OBS_LOG_MATCH_MODE: &str = "suffix";
pub const DEFAULT_OBS_LOG_SPAN_EVENTS: &str = "none";
/// Default values for observability configuration
// ### Supported Environment Values
// - `production` - Secure file-only logging
@@ -141,6 +153,7 @@ mod tests {
assert_eq!(ENV_OBS_LOG_FILENAME, "RUSTFS_OBS_LOG_FILENAME");
assert_eq!(ENV_OBS_LOG_ROTATION_TIME, "RUSTFS_OBS_LOG_ROTATION_TIME");
assert_eq!(ENV_OBS_LOG_KEEP_FILES, "RUSTFS_OBS_LOG_KEEP_FILES");
assert_eq!(ENV_OBS_LOG_SPAN_EVENTS, "RUSTFS_OBS_LOG_SPAN_EVENTS");
assert_eq!(ENV_OBS_TRACES_EXPORT_ENABLED, "RUSTFS_OBS_TRACES_EXPORT_ENABLED");
assert_eq!(ENV_OBS_METRICS_EXPORT_ENABLED, "RUSTFS_OBS_METRICS_EXPORT_ENABLED");
assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
@@ -178,6 +191,7 @@ mod tests {
assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP, "gzip");
assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD, "zstd");
assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE_PREFIX, "prefix");
assert_eq!(DEFAULT_OBS_LOG_SPAN_EVENTS, "none");
assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE, "suffix");
assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, "zstd");
}
+22
View File
@@ -0,0 +1,22 @@
[package]
name = "rustfs-connect-symbol-catalog"
version.workspace = true
edition.workspace = true
rust-version.workspace = true
license.workspace = true
publish = false
[dependencies]
hex-simd.workspace = true
object = { workspace = true, features = ["read", "std", "elf"] }
serde = { workspace = true, features = ["derive"] }
serde_json.workspace = true
sha2.workspace = true
# Match pyroscope's pprof-rs frame normalization exactly.
symbolic-demangle = { version = "=13.9.0", default-features = false, features = ["rust"] }
[dev-dependencies]
object = { workspace = true, features = ["write", "elf"] }
[lints]
workspace = true
+284
View File
@@ -0,0 +1,284 @@
// Copyright 2024 RustFS Team
// SPDX-License-Identifier: Apache-2.0
//! Release tooling only: derive reviewed CPU symbol candidates from the final ELF.
use object::{Object, ObjectSymbol};
use serde::Serialize;
use sha2::{Digest, Sha256};
use std::collections::BTreeSet;
use std::fs::{self, OpenOptions};
use std::io::{self, Read, Write};
use std::path::Path;
const MAX_BINARY_BYTES: u64 = 1_073_741_824;
const MAX_CATALOG_BYTES: usize = 32 * 1024 * 1024;
const MAX_SYMBOLS: usize = 200_000;
const NORMALIZATION: &str = "pyroscope-2.1.1-symbolic-demangle-13.9.0";
#[derive(Serialize)]
#[serde(rename_all = "camelCase")]
struct Catalog<'a> {
schema_version: u8,
normalization: &'static str,
source_commit: &'a str,
executable_sha256: String,
os_family: &'static str,
architecture: &'static str,
build_features: Vec<String>,
symbols: BTreeSet<String>,
}
fn invalid(message: &'static str) -> io::Error {
io::Error::new(io::ErrorKind::InvalidData, message)
}
fn symbol_name(raw: &[u8]) -> Option<String> {
// This is the same lossy UTF-8 conversion and demangler used by pprof-rs.
let name = symbolic_demangle::demangle(&String::from_utf8_lossy(raw)).into_owned();
if name.is_empty()
|| name.len() > 4096
|| matches!(name.as_str(), "Unknown" | "<unresolved>")
|| name.bytes().any(|b| b < 32 || b == 127)
{
return None;
}
Some(name)
}
fn catalog(binary: &[u8], source: &str, target: &str, features: Vec<String>) -> io::Result<Vec<u8>> {
if source.len() != 40 || !source.bytes().all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) {
return Err(invalid("source commit must be a lowercase full SHA"));
}
let expected_arch = match target {
"x86_64-unknown-linux-gnu" => object::Architecture::X86_64,
"aarch64-unknown-linux-gnu" => object::Architecture::Aarch64,
_ => return Err(invalid("only supported Linux GNU CPU profiling targets have catalogs")),
};
if features.is_empty()
|| features.len() > 64
|| !features.iter().any(|f| f == "pyroscope")
|| features.iter().any(|f| {
f.is_empty()
|| f.len() > 64
|| !f.as_bytes().first().is_some_and(u8::is_ascii_lowercase)
|| !f
.bytes()
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-' || b == b'_')
})
|| features.windows(2).any(|pair| pair[0] >= pair[1])
{
return Err(invalid("build features must be sorted, unique and include pyroscope"));
}
let file = object::File::parse(binary).map_err(|_| invalid("cannot parse executable"))?;
if file.format() != object::BinaryFormat::Elf
|| file.architecture() != expected_arch
|| !matches!(file.kind(), object::ObjectKind::Executable | object::ObjectKind::Dynamic)
{
return Err(invalid("the executable does not match the Linux target"));
}
if file.symbol_table().is_none() {
return Err(invalid("the executable is stripped; retain its function symbol table"));
}
let mut symbols = BTreeSet::new();
let mut symbol_bytes = 0;
for symbol in file.symbols().chain(file.dynamic_symbols()) {
if symbol.kind() != object::SymbolKind::Text || symbol.is_undefined() {
continue;
}
if let Some(name) = symbol.name_bytes().ok().and_then(symbol_name)
&& !symbols.contains(&name)
{
symbol_bytes += name.len() + 3;
if symbol_bytes > MAX_CATALOG_BYTES {
return Err(invalid("symbol catalog exceeds its byte limit"));
}
symbols.insert(name);
}
if symbols.len() > MAX_SYMBOLS {
return Err(invalid("symbol catalog exceeds its reviewed symbol limit"));
}
}
if symbols.is_empty() {
return Err(invalid("the executable has no usable function symbols"));
}
let catalog = Catalog {
schema_version: 1,
normalization: NORMALIZATION,
source_commit: source,
executable_sha256: hex_simd::encode_to_string(Sha256::digest(binary), hex_simd::AsciiCase::Lower),
os_family: "LINUX",
architecture: if expected_arch == object::Architecture::X86_64 {
"x86_64"
} else {
"aarch64"
},
build_features: features,
symbols,
};
let json = serde_json::to_vec(&catalog).map_err(io::Error::other)?;
if json.len() > MAX_CATALOG_BYTES {
return Err(invalid("symbol catalog exceeds its byte limit"));
}
Ok(json)
}
fn main() -> io::Result<()> {
let args: Vec<_> = std::env::args().skip(1).collect();
let [binary_path, source, target, features_json, output] = args.as_slice() else {
return Err(invalid(
"usage: rustfs-connect-symbol-catalog BINARY SOURCE_SHA TARGET FEATURES_JSON OUTPUT",
));
};
let input = fs::File::open(binary_path)?;
if !input.metadata()?.is_file() || input.metadata()?.len() > MAX_BINARY_BYTES {
return Err(invalid("input must be a bounded regular executable"));
}
let mut binary = Vec::new();
input.take(MAX_BINARY_BYTES + 1).read_to_end(&mut binary)?;
if u64::try_from(binary.len()).map_err(io::Error::other)? > MAX_BINARY_BYTES {
return Err(invalid("executable grew beyond its byte limit"));
}
let features = serde_json::from_str(features_json).map_err(io::Error::other)?;
let json = catalog(&binary, source, target, features)?;
// Never overwrite another build's reviewed artifact.
let mut file = OpenOptions::new().write(true).create_new(true).open(Path::new(output))?;
file.write_all(&json)?;
file.sync_all()?;
println!("{}", hex_simd::encode_to_string(Sha256::digest(&json), hex_simd::AsciiCase::Lower));
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
fn executable() -> Vec<u8> {
let mut object =
object::write::Object::new(object::BinaryFormat::Elf, object::Architecture::X86_64, object::Endianness::Little);
let section = object.section_id(object::write::StandardSection::Text);
object.append_section_data(section, &[0xc3], 1);
object.add_symbol(object::write::Symbol {
name: b"_ZN4test4main17h0123456789abcdefE".to_vec(),
value: 0,
size: 1,
kind: object::SymbolKind::Text,
scope: object::SymbolScope::Linkage,
weak: false,
section: object::write::SymbolSection::Section(section),
flags: object::SymbolFlags::None,
});
let mut bytes = object.write().expect("write ELF symbol fixture");
// The object writer emits ET_REL; this fixture models the final ET_EXEC header.
bytes[16..18].copy_from_slice(&2_u16.to_le_bytes());
bytes
}
fn bulk_executable(count: usize, suffix_len: usize) -> Vec<u8> {
let mut object =
object::write::Object::new(object::BinaryFormat::Elf, object::Architecture::X86_64, object::Endianness::Little);
let section = object.section_id(object::write::StandardSection::Text);
object.append_section_data(section, &[0xc3], 1);
let suffix = "x".repeat(suffix_len);
for index in 0..count {
object.add_symbol(object::write::Symbol {
name: format!("rustfs::profile::function_{index:06}_{suffix}").into_bytes(),
value: 0,
size: 1,
kind: object::SymbolKind::Text,
scope: object::SymbolScope::Linkage,
weak: false,
section: object::write::SymbolSection::Section(section),
flags: object::SymbolFlags::None,
});
}
let mut binary = object.write().expect("write bulk ELF fixture");
binary[16..18].copy_from_slice(&2_u16.to_le_bytes());
binary
}
#[test]
fn catalog_binds_the_exact_executable_and_normalizes_function_names() {
let binary = executable();
let encoded =
catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()]).expect("generate catalog");
let value: serde_json::Value = serde_json::from_slice(&encoded).expect("catalog JSON");
assert_eq!(value["symbols"], serde_json::json!(["test::main"]));
assert_eq!(
value["executableSha256"],
hex_simd::encode_to_string(Sha256::digest(&binary), hex_simd::AsciiCase::Lower)
);
assert_eq!(value["normalization"], NORMALIZATION);
assert!(catalog(&binary, &"a".repeat(40), "aarch64-unknown-linux-gnu", vec!["pyroscope".into()]).is_err());
assert!(catalog(&binary, "bad", "x86_64-unknown-linux-gnu", vec!["pyroscope".into()]).is_err());
assert!(catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into(); 2]).is_err());
}
#[test]
fn stripped_executables_and_wrong_abis_are_refused() {
let mut binary = executable();
// Removing section headers models a fully stripped executable without dynsym.
binary[40..48].fill(0);
binary[60..64].fill(0);
let error = catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()])
.expect_err("stripped binary must not yield an empty but trusted catalog");
assert!(error.to_string().contains("stripped"));
assert!(catalog(&executable(), &"a".repeat(40), "x86_64-unknown-linux-musl", vec!["pyroscope".into()]).is_err());
}
#[test]
fn symbol_hash_matches_the_connect_nonce_vector() {
let nonce: Vec<u8> = (0..32).collect();
let name = symbol_name(b"core::hint::black_box").expect("reviewed function name");
let mut digest = Sha256::new();
digest.update(b"rustfs-connect-cpu-symbol-v1\0");
digest.update(nonce);
digest.update(name.as_bytes());
assert_eq!(
hex_simd::encode_to_string(digest.finalize(), hex_simd::AsciiCase::Lower),
"5d5b472a0dd09124bfedf0bd243f88d5e61952b8da15d7463da019137320d77c"
);
}
#[test]
fn unresolved_and_non_function_material_cannot_become_catalog_entries() {
assert!(symbol_name(b"Unknown").is_none());
assert!(symbol_name(b"<unresolved>").is_none());
assert!(symbol_name(b"secret\nname").is_none());
assert!(symbol_name(&vec![b'x'; 4097]).is_none());
assert!(catalog(b"not ELF", &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()]).is_err());
}
#[test]
fn release_sized_catalog_retains_all_function_names() {
let binary = bulk_executable(110_000, 160);
let encoded = catalog(&binary, &"a".repeat(40), "x86_64-unknown-linux-gnu", vec!["pyroscope".into()])
.expect("retain complete bounded symbol catalog");
assert!(encoded.len() > 16 * 1024 * 1024);
assert!(encoded.len() <= MAX_CATALOG_BYTES);
let value: serde_json::Value = serde_json::from_slice(&encoded).expect("catalog JSON");
assert_eq!(value["symbols"].as_array().expect("symbol array").len(), 110_000);
}
#[test]
fn catalogs_over_either_resource_limit_are_refused() {
let source = "a".repeat(40);
let too_many = catalog(
&bulk_executable(MAX_SYMBOLS + 1, 0),
&source,
"x86_64-unknown-linux-gnu",
vec!["pyroscope".into()],
)
.expect_err("too many function names must be refused");
assert!(too_many.to_string().contains("reviewed symbol limit"));
let too_large = catalog(
&bulk_executable(10_000, 3_500),
&source,
"x86_64-unknown-linux-gnu",
vec!["pyroscope".into()],
)
.expect_err("oversized catalog must be refused");
assert!(too_large.to_string().contains("byte limit"));
}
}
+3 -3
View File
@@ -384,17 +384,17 @@ pub(crate) fn is_cluster_heal_coordination_unavailable(error: &(dyn std::error::
message.contains("500 Internal Server Error") && message.contains("cluster heal coordination unavailable")
}
/// Wait for the restarted cluster to admit the first root heal request.
/// Wait for the restarted cluster to admit a root heal request and return its response.
pub(crate) async fn start_root_heal_when_control_ready(
heal_url: &str,
heal_body: &str,
access_key: &str,
secret_key: &str,
) -> ChaosResult<()> {
) -> ChaosResult<String> {
let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(45);
loop {
match signed_admin_post(heal_url, Some(heal_body), access_key, secret_key).await {
Ok(_) => return Ok(()),
Ok(body) => return Ok(body),
Err(error) if is_cluster_heal_coordination_unavailable(error.as_ref()) && tokio::time::Instant::now() < deadline => {
tokio::time::sleep(std::time::Duration::from_millis(250)).await;
}
+84 -3
View File
@@ -22,6 +22,8 @@
//! - Common test constants and utilities
use aws_sdk_s3::config::{Credentials, Region};
use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
use aws_sdk_s3::operation::list_buckets::ListBucketsError;
use aws_sdk_s3::{Client, Config};
use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
use http::header::{CONTENT_TYPE, HOST};
@@ -61,6 +63,41 @@ const TEST_PORT_COUNTER_PATH: &str = "/tmp/rustfs_e2e_next_port";
const TEST_PORT_LOCK_DIR: &str = "/tmp/rustfs_e2e_port_allocator.lock";
const TEST_PORT_LOCK_STALE_AFTER: Duration = Duration::from_secs(30);
fn list_buckets_readiness_error(err: &SdkError<ListBucketsError>) -> String {
// SDK Display reports only the category. Do not expose raw response bodies or headers.
let service_code = err.as_service_error().and_then(|error| error.code()).filter(|code| {
matches!(
*code,
"ServiceUnavailable"
| "ServerNotInitialized"
| "InternalError"
| "AccessDenied"
| "InvalidAccessKeyId"
| "SignatureDoesNotMatch"
)
});
let transport = match err {
SdkError::DispatchFailure(failure) if failure.is_timeout() => Some("timeout"),
SdkError::DispatchFailure(failure) if failure.is_io() => Some("io"),
SdkError::DispatchFailure(failure) if failure.is_user() => Some("user"),
SdkError::DispatchFailure(failure) if failure.is_other() => Some("other"),
_ => None,
};
let mut source: Option<&(dyn std::error::Error + 'static)> = Some(err);
let mut io_error = None;
while let Some(error) = source {
if let Some(error) = error.downcast_ref::<std::io::Error>() {
io_error = Some((error.kind(), error.raw_os_error()));
break;
}
source = error.source();
}
format!(
"{err}; http_status={:?}; service_code={service_code:?}; transport={transport:?}; io_error={io_error:?}",
err.raw_response().map(|response| response.status().as_u16())
)
}
fn capture_log_path(log_dir: &Path, temp_dir: &str) -> Option<PathBuf> {
let temp_name = Path::new(temp_dir).file_name()?.to_string_lossy();
Some(log_dir.join(format!("{temp_name}.log")))
@@ -1575,8 +1612,9 @@ impl RustFSTestClusterEnvironment {
///
/// Verifies service availability by calling the S3 `list_buckets` API against the requested node,
/// retries up to 120 times with a 1-second interval between attempts.
async fn wait_for_node_service_ready(&self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
async fn wait_for_node_service_ready(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let client = self.create_s3_client(node_idx)?;
let mut last_error = None;
for attempt in 0..120 {
match client.list_buckets().send().await {
@@ -1584,13 +1622,28 @@ impl RustFSTestClusterEnvironment {
info!("Cluster node {} service ready after {} attempts", node_idx, attempt + 1);
return Ok(());
}
Err(_) => {
Err(err) => {
last_error = Some(err);
sleep(Duration::from_secs(1)).await;
}
}
}
Err(format!("Cluster node {} service failed to become ready", node_idx).into())
let last_error = last_error.as_ref().map(list_buckets_readiness_error);
// Observe liveness only after the existing readiness budget is exhausted, before cleanup.
let process_status = match self.nodes[node_idx].process.as_mut() {
Some(process) => match process.try_wait() {
Ok(Some(status)) => format!("exited ({status})"),
Ok(None) => "running".to_string(),
Err(error) => format!("unavailable (kind={:?}, os_code={:?})", error.kind(), error.raw_os_error()),
},
None => "not started".to_string(),
};
Err(format!(
"Cluster node {node_idx} service failed to become ready; last ListBuckets error={last_error:?}; process_status={process_status}; capture_log_path={:?}",
self.node_capture_log_paths[node_idx]
)
.into())
}
/// Create an S3 client configured to communicate with a specific cluster node.
@@ -2101,6 +2154,34 @@ mod tests {
verify_awscurl_path(&executable).expect("an available client with a working help command should pass");
}
#[test]
fn readiness_error_reports_transport_without_sensitive_source_text() {
use aws_sdk_s3::error::ConnectorError;
let sensitive = "request credentials must not appear in diagnostics";
for (connector, category) in [
(
ConnectorError::io(std::io::Error::new(ErrorKind::ConnectionReset, sensitive).into()),
"io",
),
(ConnectorError::timeout(sensitive.into()), "timeout"),
(ConnectorError::user(sensitive.into()), "user"),
(ConnectorError::other(sensitive.into(), None), "other"),
] {
let diagnostic = list_buckets_readiness_error(&SdkError::dispatch_failure(connector));
assert!(diagnostic.contains(&format!("transport=Some(\"{category}\")")), "{diagnostic}");
assert!(diagnostic.contains("http_status=None"), "{diagnostic}");
assert!(!diagnostic.contains(sensitive), "{diagnostic}");
if category == "io" {
assert!(diagnostic.contains("io_error=Some((ConnectionReset, None))"), "{diagnostic}");
}
}
let os_error = std::io::Error::from_raw_os_error(13);
let expected = format!("io_error=Some(({:?}, Some(13)))", os_error.kind());
let diagnostic = list_buckets_readiness_error(&SdkError::dispatch_failure(ConnectorError::io(os_error.into())));
assert!(diagnostic.contains(&expected), "{diagnostic}");
}
#[test]
fn capture_log_path_uses_temp_directory_basename() {
assert_eq!(
@@ -298,7 +298,11 @@ mod tests {
assert_eq!(empty_head_resp.content_disposition(), None);
assert_eq!(empty_head_resp.content_encoding(), None);
assert_eq!(empty_head_resp.content_language(), None);
assert_eq!(empty_head_resp.content_type(), None);
assert_eq!(
empty_head_resp.content_type(),
Some("binary/octet-stream"),
"REPLACE without an explicit content type must default to binary/octet-stream, matching S3"
);
assert_eq!(empty_head_resp.expires_string(), None);
assert_eq!(empty_head_resp.website_redirect_location(), None);
@@ -394,7 +398,15 @@ mod tests {
assert_eq!(head.content_disposition(), (field == "content-disposition").then_some("attachment"));
assert_eq!(head.content_encoding(), (field == "content-encoding").then_some("gzip"));
assert_eq!(head.content_language(), (field == "content-language").then_some("de"));
assert_eq!(head.content_type(), (field == "content-type").then_some("text/field"));
assert_eq!(
head.content_type(),
Some(if field == "content-type" {
"text/field"
} else {
"binary/octet-stream"
}),
"REPLACE without an explicit content type must default to binary/octet-stream, matching S3"
);
assert_eq!(
head.expires_string(),
(field == "expires").then_some(replacement_expires_http_date.as_str())
@@ -420,7 +432,11 @@ mod tests {
.send()
.await
.expect("HEAD failed for metadata collision case");
assert_eq!(collision_head.content_type(), None);
assert_eq!(
collision_head.content_type(),
Some("binary/octet-stream"),
"user-metadata under the reserved 'content-type' key must not satisfy the system content-type default"
);
assert_eq!(collision_head.content_encoding(), None);
assert_eq!(
collision_head.metadata().and_then(|metadata| metadata.get("content-type")),
@@ -282,7 +282,7 @@ async fn sdk_list_bucket_and_list_bucket_versions_permissions_are_independent()
}
#[tokio::test]
async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers() -> TestResult {
async fn explicit_force_delete_permission_removes_prefix_versions_and_delete_markers() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
@@ -300,7 +300,14 @@ async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers()
put(&root, bucket, "folder-sibling/keep.txt").await?;
let keep = versions(&root, bucket, "keep.txt").await?;
let sibling = versions(&root, bucket, "folder-sibling/").await?;
let user = policy_user(&env, "consoleAdmin", None).await?;
let user = policy_user(
&env,
"explicit-force-delete",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")}
]})),
)
.await?;
force_delete(&user, bucket, "folder/").await?;
assert!(
@@ -322,6 +329,126 @@ async fn console_admin_force_delete_removes_prefix_versions_and_delete_markers()
Ok(())
}
#[tokio::test]
async fn force_delete_header_requires_explicit_permission() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
let root = env.create_s3_client();
let admin = policy_user(&env, "consoleAdmin", None).await?;
let owner_bucket = "force-owner-bucket";
root.create_bucket().bucket(owner_bucket).send().await?;
put(&root, owner_bucket, "kept-until-force.txt").await?;
force_delete_bucket(&root, owner_bucket, "x-rustfs-force-delete").await?;
assert!(
head_bucket_missing(&root, owner_bucket).await?,
"root force-delete must remove a non-empty bucket"
);
let denied_bucket = "force-admin-denied";
root.create_bucket().bucket(denied_bucket).send().await?;
put(&root, denied_bucket, "folder/child.txt").await?;
put(&root, denied_bucket, "plain.txt").await?;
assert_boxed_denied(force_delete(&admin, denied_bucket, "folder/").await);
assert!(
!versions(&root, denied_bucket, "folder/").await?.is_empty(),
"consoleAdmin without s3:ForceDeleteObject must not recursively delete"
);
assert_boxed_denied(force_delete_bucket(&admin, denied_bucket, "x-rustfs-force-delete").await);
assert_boxed_denied(force_delete_bucket(&admin, denied_bucket, "x-minio-force-delete").await);
assert!(
!head_bucket_missing(&root, denied_bucket).await?,
"consoleAdmin without s3:ForceDeleteBucket must not force-delete a non-empty bucket"
);
let not_empty = admin.delete_bucket().bucket(denied_bucket).send().await;
let not_empty = not_empty.expect_err("plain delete of a non-empty bucket must fail");
assert_eq!(
not_empty.as_service_error().and_then(|error| error.code()),
Some("BucketNotEmpty"),
"plain delete must keep the emptiness check, got {not_empty:?}"
);
admin.delete_object().bucket(denied_bucket).key("plain.txt").send().await?;
assert!(
versions(&root, denied_bucket, "plain.txt").await?.is_empty(),
"plain DeleteObject without the force header must stay allowed for consoleAdmin"
);
let object_bucket = "force-explicit-object";
root.create_bucket().bucket(object_bucket).send().await?;
put(&root, object_bucket, "folder/child.txt").await?;
put(&root, object_bucket, "keep.txt").await?;
let keep = versions(&root, object_bucket, "keep.txt").await?;
let object_user = policy_user(
&env,
"explicit-object-force",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{object_bucket}/*")}
]})),
)
.await?;
force_delete(&object_user, object_bucket, "folder/").await?;
assert!(versions(&root, object_bucket, "folder/").await?.is_empty());
assert_eq!(versions(&root, object_bucket, "keep.txt").await?, keep);
let allowed_bucket = "force-explicit-bucket";
root.create_bucket().bucket(allowed_bucket).send().await?;
put(&root, allowed_bucket, "inside.txt").await?;
let bucket_user = policy_user(
&env,
"explicit-bucket-force",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteBucket","s3:ForceDeleteBucket"],"Resource":[
format!("arn:aws:s3:::{allowed_bucket}"), format!("arn:aws:s3:::{allowed_bucket}/*")
]}
]})),
)
.await?;
force_delete_bucket(&bucket_user, allowed_bucket, "x-minio-force-delete").await?;
assert!(
head_bucket_missing(&root, allowed_bucket).await?,
"an explicit s3:ForceDeleteBucket grant must remove a non-empty bucket"
);
Ok(())
}
async fn force_delete_bucket(
client: &Client,
bucket: &str,
header: &str,
) -> Result<(), Box<SdkError<aws_sdk_s3::operation::delete_bucket::DeleteBucketError>>> {
let header: &'static str = match header {
"x-rustfs-force-delete" => "x-rustfs-force-delete",
"x-minio-force-delete" => "x-minio-force-delete",
other => panic!("unexpected force-delete header {other}"),
};
client
.delete_bucket()
.bucket(bucket)
.customize()
.mutate_request(move |request| {
request.headers_mut().insert(header, "true");
})
.send()
.await
.map(|_| ())
.map_err(Box::new)
}
async fn head_bucket_missing(client: &Client, bucket: &str) -> TestResult<bool> {
match client.head_bucket().bucket(bucket).send().await {
Ok(_) => Ok(false),
Err(error) => {
let code = error.as_service_error().and_then(|service| service.code());
if matches!(code, Some("NotFound" | "NoSuchBucket")) {
Ok(true)
} else {
Err(error.into())
}
}
}
}
#[tokio::test]
async fn force_delete_authorizes_only_its_path_scope_without_list_permissions() -> TestResult {
init_logging();
@@ -339,7 +466,7 @@ async fn force_delete_authorizes_only_its_path_scope_without_list_permissions()
&env,
"delete-only",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":[
format!("arn:aws:s3:::{bucket}/selected.txt"), format!("arn:aws:s3:::{bucket}/selected.txt/*")
]},
{"Effect":"Deny","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/selected.txt-sibling")}
@@ -376,7 +503,7 @@ async fn force_directory_delete_cannot_remove_an_unauthorized_colliding_parent()
&env,
"parent-denier",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/collision.txt"),
"Condition":{"StringEquals":{"s3:VersionId":protected_parent_version}}}
]})),
@@ -409,7 +536,7 @@ async fn force_unversioned_directory_requires_only_delete_object() -> TestResult
&env,
"unversioned-deleter",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":"s3:DeleteObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Allow","Action":["s3:DeleteObject","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/*")}
]})),
)
@@ -438,7 +565,7 @@ async fn force_delete_denied_child_preserves_every_object_despite_bucket_allow()
&env,
"child-denier",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Deny","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ReplicateDelete"],"Resource":format!("arn:aws:s3:::{bucket}/folder/z-denied.txt")}
]})),
)
@@ -468,7 +595,7 @@ async fn force_delete_denied_child_preserves_every_object_despite_bucket_allow()
&env,
"replica-deleter",
Some(json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":"s3:DeleteObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Allow","Action":["s3:DeleteObject","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Allow","Action":"s3:ReplicateDelete","Resource":format!("arn:aws:s3:::{bucket}/*")}
]})),
)
@@ -500,7 +627,7 @@ async fn force_delete_denied_historical_version_preserves_versions_and_markers()
put(&root, bucket, "folder/a-allowed.txt").await?;
let policy = |version: &str| {
json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Deny","Action":"s3:DeleteObjectVersion","Resource":format!("arn:aws:s3:::{bucket}/folder/*"),
"Condition":{"StringEquals":{"s3:VersionId":version}}}
]})
@@ -628,7 +755,7 @@ async fn force_delete_checks_every_version_page_before_mutation() -> TestResult
.try_collect::<Vec<_>>()
.await?;
put(&root, bucket, "folder/z-denied.txt").await?;
let allow = json!({"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion"],"Resource":format!("arn:aws:s3:::{bucket}/*")});
let allow = json!({"Effect":"Allow","Action":["s3:DeleteObject","s3:DeleteObjectVersion","s3:ForceDeleteObject"],"Resource":format!("arn:aws:s3:::{bucket}/*")});
let user = policy_user(
&env,
"paged-deleter",
+219 -25
View File
@@ -34,6 +34,9 @@ use crate::common::{
};
use crate::replication_extension_test::LOOPBACK_REPLICATION_TARGET_ENV;
use aws_sdk_s3::Client;
use aws_sdk_s3::config::retry::RetryConfig;
use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
use aws_sdk_s3::operation::{get_object::GetObjectError, put_object::PutObjectError};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
use http::{Method, StatusCode};
@@ -41,7 +44,7 @@ use sha2::{Digest, Sha256};
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::time::Duration;
use tokio::time::{Instant, sleep};
use tokio::time::{Instant, sleep, timeout_at};
use uuid::Uuid;
pub(crate) type TestResult<T = ()> = Result<T, Box<dyn std::error::Error + Send + Sync>>;
@@ -400,9 +403,9 @@ pub(crate) async fn put_inventory(
Ok(inventory)
}
/// Retry only transport-level service availability failures while a data
/// movement operation changes the pool map. Generic InternalError responses
/// remain fatal because accepting them would hide server defects.
/// Retry only explicit S3 availability responses while a data movement
/// operation changes the pool map. Transport failures and InternalError
/// responses remain fatal because callers already require a ready endpoint.
pub(crate) async fn put_inventory_retrying(
client: &Client,
bucket: &str,
@@ -449,19 +452,48 @@ where
let deadline = Instant::now() + timeout;
let mut delay = Duration::from_millis(50);
loop {
let last_error = match probe().await {
Ok(true) => return Ok(()),
Ok(false) => format!("{label} still false"),
Err(error) => error.to_string(),
};
if Instant::now() >= deadline {
return Err(format!("{label} did not become true within {timeout:?}: {last_error}").into());
return Err(format!("{label} did not become true within {timeout:?}").into());
}
sleep(delay).await;
// A pending request must consume the same budget as unsuccessful probes.
match timeout_at(deadline, probe()).await {
Err(_) => return Err(format!("{label} did not become true within {timeout:?}").into()),
Ok(Err(error)) => return Err(error),
Ok(Ok(true)) => return Ok(()),
Ok(Ok(false)) => {}
}
tokio::time::sleep_until((Instant::now() + delay).min(deadline)).await;
delay = (delay * 2).min(Duration::from_secs(1));
}
}
// Boxed SDK errors retain the operation's service code; Display only says
// "service error" and cannot distinguish convergence from a server defect.
pub(crate) fn s3_probe_error_code<'a>(error: &'a (dyn std::error::Error + Send + Sync + 'static)) -> Option<&'a str> {
error
.downcast_ref::<SdkError<GetObjectError>>()
.and_then(|error| error.as_service_error())
.and_then(ProvideErrorMetadata::code)
.or_else(|| {
error
.downcast_ref::<SdkError<PutObjectError>>()
.and_then(|error| error.as_service_error())
.and_then(ProvideErrorMetadata::code)
})
}
pub(crate) fn s3_probe_client(client: &Client) -> Client {
// The probe owns the retry policy. SDK retries must not hide a response
// that the probe would classify as fatal.
Client::from_conf(
client
.config()
.to_builder()
.retry_config(RetryConfig::standard().with_max_attempts(1))
.build(),
)
}
pub(crate) async fn cluster_admin(
cluster: &RustFSTestClusterEnvironment,
method: Method,
@@ -598,15 +630,20 @@ pub(crate) async fn wait_for_replicated_bytes(
expected: &[u8],
timeout: Duration,
) -> TestResult {
let client = s3_probe_client(client);
wait_until(
timeout,
|| async {
match get_object_bytes(client, bucket, key).await {
match get_object_bytes(&client, bucket, key).await {
Ok(got) if got.as_slice() == expected => Ok(true),
Ok(_) => Ok(false),
Ok(got) => Err(format!(
"replicated object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
sha256_hex(expected),
sha256_hex(&got)
)
.into()),
Err(error) => {
let message = error.to_string();
if message.contains("NoSuchKey") || message.contains("NotFound") {
if matches!(s3_probe_error_code(error.as_ref()), Some("NoSuchKey" | "NotFound")) {
Ok(false)
} else {
Err(error)
@@ -988,6 +1025,7 @@ pub(crate) async fn list_pools_json(cluster: &RustFSTestClusterEnvironment) -> T
}
pub(crate) async fn retrying_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, timeout: Duration) -> TestResult {
let client = s3_probe_client(client);
wait_until(
timeout,
|| {
@@ -999,8 +1037,7 @@ pub(crate) async fn retrying_put(client: &Client, bucket: &str, key: &str, body:
match put_object(&client, &bucket, &key, body).await {
Ok(()) => Ok(true),
Err(error) => {
let message = error.to_string();
if message.contains("SlowDown") || message.contains("ServiceUnavailable") || message.contains("503") {
if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
Ok(false)
} else {
Err(error)
@@ -1021,19 +1058,20 @@ pub(crate) async fn retrying_get_equals(
expected: &[u8],
timeout: Duration,
) -> TestResult {
let client = s3_probe_client(client);
wait_until(
timeout,
|| async {
match get_object_bytes(client, bucket, key).await {
match get_object_bytes(&client, bucket, key).await {
Ok(got) if got.as_slice() == expected => Ok(true),
Ok(_) => Ok(false),
Ok(got) => Err(format!(
"object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
sha256_hex(expected),
sha256_hex(&got)
)
.into()),
Err(error) => {
let message = error.to_string();
if message.contains("NoSuchKey")
|| message.contains("SlowDown")
|| message.contains("ServiceUnavailable")
|| message.contains("503")
{
if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
Ok(false)
} else {
Err(error)
@@ -1046,6 +1084,162 @@ pub(crate) async fn retrying_get_equals(
.await
}
#[cfg(test)]
mod retry_tests {
use super::*;
use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction, Operation, SeedMetadata};
use std::cell::Cell;
fn client(target: &FakeS3Target) -> Client {
Client::from_conf(build_test_s3_config(
target.endpoint(),
FAKE_ACCESS_KEY,
FAKE_SECRET_KEY,
None,
"distributed-retry-test",
))
}
#[tokio::test]
async fn wait_until_preserves_first_error() {
let attempts = Cell::new(0);
let error = wait_until(
Duration::from_secs(5),
|| {
attempts.set(attempts.get() + 1);
std::future::ready(if attempts.get() == 1 {
Err(std::io::Error::new(std::io::ErrorKind::PermissionDenied, "permanent probe failure").into())
} else {
Ok(true)
})
},
"permanent failure",
)
.await
.expect_err("a later success must not hide a permanent failure");
assert_eq!(attempts.get(), 1);
assert_eq!(
error
.downcast_ref::<std::io::Error>()
.expect("preserve the original error")
.kind(),
std::io::ErrorKind::PermissionDenied
);
}
#[tokio::test]
async fn wait_until_retries_explicit_pending_state() -> TestResult {
let attempts = Cell::new(0);
wait_until(
Duration::from_secs(5),
|| {
attempts.set(attempts.get() + 1);
std::future::ready(Ok(attempts.get() == 2))
},
"eventual readiness",
)
.await?;
assert_eq!(attempts.get(), 2);
Ok(())
}
#[tokio::test]
async fn wait_until_bounds_a_pending_probe() {
let error = tokio::time::timeout(
Duration::from_secs(5),
wait_until(Duration::from_millis(10), std::future::pending, "hung probe"),
)
.await
.expect("the probe deadline must finish before the test watchdog")
.expect_err("a permanently pending probe must time out");
assert!(error.to_string().contains("hung probe did not become true"));
}
#[tokio::test]
async fn wait_until_does_not_start_a_probe_after_its_deadline() {
let attempts = Cell::new(0);
wait_until(
Duration::ZERO,
|| {
attempts.set(attempts.get() + 1);
std::future::ready(Ok(true))
},
"expired budget",
)
.await
.expect_err("an expired budget must not admit another probe");
assert_eq!(attempts.get(), 0);
}
#[tokio::test]
async fn retrying_put_does_not_hide_internal_error() -> TestResult {
let target = FakeS3Target::start().await?;
target.create_bucket("retry-probe");
target.inject(Operation::PutObject, FaultAction::ResponseStatus(500), 1);
let error = retrying_put(&client(&target), "retry-probe", "key", b"body".to_vec(), Duration::from_secs(5))
.await
.expect_err("InternalError must remain fatal even if the next PUT would succeed");
assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
assert_eq!(target.requests().len(), 1);
Ok(())
}
#[tokio::test]
async fn retrying_put_allows_transient_availability_errors() -> TestResult {
let target = FakeS3Target::start().await?;
target.create_bucket("retry-probe");
for status in [429, 503] {
target.inject(Operation::PutObject, FaultAction::ResponseStatus(status), 1);
retrying_put(&client(&target), "retry-probe", "key", b"body".to_vec(), Duration::from_secs(5)).await?;
assert_eq!(target.take_requests().len(), 2);
}
Ok(())
}
#[tokio::test]
async fn object_polls_reject_wrong_object_bytes() -> TestResult {
let target = FakeS3Target::start().await?;
target.create_bucket("retry-probe");
target.put_seed_object("retry-probe", "key", "wrong bytes", &SeedMetadata::default());
let error = retrying_get_equals(&client(&target), "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
.await
.expect_err("a successful response with wrong bytes must fail immediately");
assert!(error.to_string().contains("bytes mismatch"));
assert_eq!(target.take_requests().len(), 1);
let error = wait_for_replicated_bytes(&client(&target), "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
.await
.expect_err("replication may lag, but it must not return corrupt bytes");
assert!(error.to_string().contains("bytes mismatch"));
assert_eq!(target.requests().len(), 1);
Ok(())
}
#[tokio::test]
async fn object_polls_distinguish_committed_and_replicated_keys() -> TestResult {
let target = FakeS3Target::start().await?;
target.create_bucket("retry-probe");
target.put_seed_object("retry-probe", "key", "expected bytes", &SeedMetadata::default());
let client = client(&target);
for (status, expected_code) in [(404, "NoSuchKey"), (500, "InternalError"), (403, "AccessDenied")] {
target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
let error = retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5))
.await
.expect_err("an acknowledged object must not disappear or fail before a later successful read");
assert_eq!(s3_probe_error_code(error.as_ref()), Some(expected_code));
assert_eq!(target.take_requests().len(), 1);
}
for status in [429, 503] {
target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await?;
assert_eq!(target.take_requests().len(), 2);
}
target.inject(Operation::GetObject, FaultAction::ResponseStatus(404), 1);
wait_for_replicated_bytes(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await?;
assert_eq!(target.requests().len(), 2);
Ok(())
}
}
#[tokio::test]
async fn append_single_node_pool_extends_ellipses_volumes() {
let mut env = RustFSTestClusterEnvironment::with_topology(ClusterTopology::per_node_pools(2, vec![vec![0], vec![1]]))
@@ -184,7 +184,7 @@ async fn four_node_health_inventory_metrics_and_audit_delivery_are_consistent()
let (status, metrics_body) = admin_request(
&node.url,
Method::GET,
"/rustfs/admin/v3/metrics?n=1&by-host=true&by-disk=true",
"/rustfs/admin/v3/realtime?n=1&by-host=true&by-disk=true",
None,
&dist.cluster.access_key,
&dist.cluster.secret_key,
@@ -257,7 +257,7 @@ async fn node_admin_body(dist: &DistCluster, node: usize, path: &str) -> TestRes
}
async fn http_put_counts(dist: &DistCluster, node: usize) -> TestResult<[u64; 2]> {
let body = node_admin_body(dist, node, "/rustfs/admin/v3/metrics?types=512&by-host=true&n=1").await?;
let body = node_admin_body(dist, node, "/rustfs/admin/v3/realtime?types=512&by-host=true&n=1").await?;
let sample: RealtimeMetrics = serde_json::from_str(body.lines().next().ok_or("empty HTTP metrics stream")?)?;
assert!(sample.errors.is_empty(), "HTTP metrics returned errors: {:?}", sample.errors);
let http = sample.aggregated.http.ok_or("HTTP metrics missing at WARN log level")?;
@@ -26,7 +26,7 @@
use super::harness::{
DistCluster, DistLayout, TestResult, assert_object_bytes, cluster_admin_ok, enable_versioning, get_object_bytes, put_object,
unique_bucket, wait_until,
s3_probe_client, s3_probe_error_code, sha256_hex, unique_bucket, wait_until,
};
use crate::common::{
AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, init_logging,
@@ -111,6 +111,7 @@ async fn create_iam_user(dist: &DistCluster, user: &str, secret: &str, policy_na
}
async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, label: &str) -> TestResult {
let client = s3_probe_client(client);
wait_until(
CREDENTIAL_TIMEOUT,
|| {
@@ -119,8 +120,18 @@ async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, l
let key = key.to_string();
let body = body.clone();
async move {
put_object(&client, &bucket, &key, body).await?;
Ok(true)
match put_object(&client, &bucket, &key, body).await {
Ok(()) => Ok(true),
Err(error)
if matches!(
s3_probe_error_code(error.as_ref()),
Some("AccessDenied" | "InvalidAccessKeyId" | "SlowDown" | "ServiceUnavailable")
) =>
{
Ok(false)
}
Err(error) => Err(error),
}
}
},
label,
@@ -129,6 +140,7 @@ async fn wait_for_put(client: &Client, bucket: &str, key: &str, body: Vec<u8>, l
}
async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8], label: &str) -> TestResult {
let client = s3_probe_client(client);
wait_until(
CREDENTIAL_TIMEOUT,
|| {
@@ -137,8 +149,24 @@ async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8
let key = key.to_string();
let expected = expected.to_vec();
async move {
let got = get_object_bytes(&client, &bucket, &key).await?;
Ok(got == expected)
match get_object_bytes(&client, &bucket, &key).await {
Ok(got) if got == expected => Ok(true),
Ok(got) => Err(format!(
"{label}: object {bucket}/{key} bytes mismatch: expected sha256={} got sha256={}",
sha256_hex(&expected),
sha256_hex(&got)
)
.into()),
Err(error)
if matches!(
s3_probe_error_code(error.as_ref()),
Some("AccessDenied" | "InvalidAccessKeyId" | "SlowDown" | "ServiceUnavailable")
) =>
{
Ok(false)
}
Err(error) => Err(error),
}
}
},
label,
@@ -146,6 +174,57 @@ async fn wait_for_bytes(client: &Client, bucket: &str, key: &str, expected: &[u8
.await
}
#[cfg(test)]
mod retry_tests {
use super::*;
use crate::common::build_test_s3_config;
use crate::fake_s3_target::{FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction, Operation};
#[tokio::test]
async fn iam_polls_only_retry_auth_propagation_and_availability() -> TestResult {
let target = FakeS3Target::start().await?;
target.create_bucket("upgrade-retry-probe");
let client = Client::from_conf(build_test_s3_config(
target.endpoint(),
FAKE_ACCESS_KEY,
FAKE_SECRET_KEY,
None,
"upgrade-retry-test",
));
for status in [403, 503] {
target.inject(Operation::PutObject, FaultAction::ResponseStatus(status), 1);
wait_for_put(&client, "upgrade-retry-probe", "key", b"body".to_vec(), "IAM PUT").await?;
assert_eq!(target.take_requests().len(), 2);
target.inject(Operation::GetObject, FaultAction::ResponseStatus(status), 1);
wait_for_bytes(&client, "upgrade-retry-probe", "key", b"body", "IAM GET").await?;
assert_eq!(target.take_requests().len(), 2);
}
target.inject(Operation::PutObject, FaultAction::ResponseStatus(500), 1);
let error = wait_for_put(&client, "upgrade-retry-probe", "key", b"body".to_vec(), "IAM PUT")
.await
.expect_err("IAM convergence must not hide InternalError on PUT");
assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
assert_eq!(target.take_requests().len(), 1);
target.inject(Operation::GetObject, FaultAction::ResponseStatus(500), 1);
let error = wait_for_bytes(&client, "upgrade-retry-probe", "key", b"body", "IAM GET")
.await
.expect_err("IAM convergence must not hide InternalError on GET");
assert_eq!(s3_probe_error_code(error.as_ref()), Some("InternalError"));
assert_eq!(target.take_requests().len(), 1);
let error = wait_for_bytes(&client, "upgrade-retry-probe", "key", b"wrong bytes", "IAM GET")
.await
.expect_err("IAM convergence must not hide a payload mismatch");
assert!(error.to_string().contains("bytes mismatch"));
assert_eq!(target.requests().len(), 1);
Ok(())
}
}
async fn seed_history_and_iam(dist: &DistCluster) -> TestResult<UpgradeSeed> {
let history_bucket = unique_bucket("upg-hist");
let versioned_bucket = unique_bucket("upg-ver");

Some files were not shown because too many files have changed in this diff Show More