Compare commits

..

67 Commits

Author SHA1 Message Date
overtrue 2a5d4d882a feat(gateway-adapter): add IAM-backed CredentialProvider
Add the ring-2 crate rustfs-gateway-adapter with IamCredentials, the
gateway CredentialProvider answered by RustFS IAM (rustfs/backlog#2760,
credentials half; PolicySource is deferred).

- Pin rustfs-gateway at the existing gateway rev with
  default-features = false so the server crate stays out of the build.
- IamKeyLookup is a narrow IAM trait implemented for IamSys so the
  adapter is testable without an in-memory IAM backend.
- Root credentials are injected at construction and resolve before IAM
  is ready.
- Mapping: keystone: prefix -> NotFound (deliberate change from the
  legacy empty secret); IAM not ready or not initialized -> Unavailable;
  unknown key or NoSuch{User,ServiceAccount,TempAccount,Account} ->
  NotFound; other IAM errors -> Backend; invalid or "off" principals
  resolve disabled; STS credentials carry a SessionBinding, and an STS
  credential without an expiry is a Backend fault rather than a
  never-ending session.
- No authorization in lookup; no Debug on secret-holding types; IAM
  error text is dropped after classification.
2026-10-30 00:03:40 +08:00
overtrue 279a88ab56 chore(gateway): merge main into integration 2026-10-29 20:19:36 +08:00
overtrue b4abe64ada chore(gateway): merge main into integration
Sync main 84e24e767 into gateway/integration. Ports main-side additions to
the integration boundaries: the tier deletion diagnostics admin handlers move
to the owned admin request/response types, and the new lifecycle rule checks
in ecstore use the persisted lifecycle types instead of s3s DTOs. Keeps the
gateway git source in deny.toml while dropping the smithy-rs pin main retired,
and refreshes the test-count baseline for the merged tree.
2026-10-29 16:01:13 +08:00
overtrue add363f084 chore(gateway): merge main into integration
Sync main 9601a7c4b into gateway/integration. Ports main-side additions to the
owned admin boundary (plugin delegation/attempt handlers, notification target
edit tests), drops a new ecstore s3s::dto reference in favour of the persisted
lifecycle filter, and refreshes the e2e selection digests and test-count
baseline for the merged tree.
2026-10-29 07:31:24 +08:00
overtrue 1f2dbe36e5 test(ecstore): pin native history on cancellation under the cancellable queue
main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.
2026-10-27 22:08:04 +08:00
overtrue 67fbafa13e chore(gateway): merge main into integration
Source: main 4e48c390e7
Target: gateway/integration 1000860cb8

Conflicts resolved, keeping both sides:
- crates/ecstore/src/disk/{fs,local}.rs: main's batched rollback writes,
  unified blocking write path and relocated open_file, with integration's
  native publication records restored at each publication point and the
  publication_history field kept beside main's multipart proof authority.
- crates/ecstore/src/set_disk/ops/multipart.rs: union of the imports.
- rustfs/src/storage/rpc/node_service.rs: main's MRF lifecycle reader
  capability branch; the heal capability keeps integration's restore
  copy-back probe.
- rustfs/src/admin/handlers/{audit,event,plugins_instances}.rs: main's
  target_read inventory snapshot on integration's owned request boundary;
  the new event.rs inventory test now builds owned requests.
- scripts/check_s3s_footprint.sh: integration's baseline-file version.
- .config/e2e-{full,smoke}-selection.txt: digests recomputed over the
  union of both heads' CI listings (full 700 darwin / 708 linux, smoke 232).

Also migrated main's new rebalance delete-all expiry test to the persisted
lifecycle types integration uses in ecstore, and raised the test-count
baseline to the merged counts (every decrease comes from main).
2026-10-27 08:38:17 +08:00
Chris An 1000860cb8 refactor(admin): drop the legacy target descriptor JSON response helper (#8968)
`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.
2026-10-27 00:15:37 +08:00
Chris An b783f404ad refactor(admin): migrate identity and cluster handlers to owned requests (#8966)
* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.
2026-10-26 22:23:47 +08:00
Chris An 11a991c777 refactor(admin): complete the owned admin request boundary (#8949)
* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes
2026-10-26 17:04:02 +08:00
Chris An 3f2b6f690a refactor(admin): migrate health and prefix usage to owned requests (#8887)
* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request
2026-10-26 07:34:31 +08:00
overtrue 88b626e7a0 chore(gateway): merge main into integration
Source: main 0f24511535
Target: gateway/integration 8cf6187ff4

Resolve the e2e-full selection digest conflict by recomputing both
platform digests over the merged nextest listing (685 darwin / 693
linux): main adds two rpc_fault_proxy unit tests, integration keeps its
dual-stack differential and anonymous-access additions. Raise the rustfs
test-count baseline row to the merged count (5932).
2026-10-25 09:58:02 +08:00
overtrue 8cf6187ff4 chore(gateway): merge main into integration 2026-10-25 01:50:03 +08:00
Chris a0a0cee9eb refactor(auth): derive policy conditions from AuthzFacts (#8442)
* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline
2026-10-24 16:56:01 +09:00
overtrue 80fff4b5c5 Merge gateway/integration (#8844) into the main sync
Only the s3s footprint baseline conflicted; s3_error_lines tightens to 289.
2026-10-24 09:30:22 +08:00
Chris d91dfcedd2 refactor(s3): migrate private multipart validation errors (#8844) 2026-10-24 09:17:48 +09:00
overtrue 79c4fedf8c Merge gateway/integration into the main sync
Brings in #8825, #8831, #8836 and #8839. Conflicts were import lists and test blocks (both sides kept), plus the new restore_copyback request flag next to main's data-movement field. Main replaced completed_multipart_object_part with prepare_complete_multipart_parts, so the stale import is dropped.

Main's xl.meta fast-path scanner (inspect_v2_object_slice) predates #8836 and classified a transitioned version with a pending v1 restore copyback as having no physical data dir, which made find_unshared_data_dir_for_version drop the copyback data dir. The scanner now reads the copyback marker with get_consistent_bytes semantics (both aliases, ASCII case-insensitive, empty or conflicting values count as absent), and uses_data_dir keeps main's allocation-free transition check plus #8836's copyback exclusion. find_unshared_data_dir_for_version_keeps_pending_copyback_data_dir failed on the plain merge and passes now.
2026-10-24 07:51:55 +08:00
Chris 73c846e09c refactor(storage): migrate option errors to domain types (#8839) 2026-10-24 05:50:10 +09:00
Chris 6938c467d0 fix(ecstore): finalize restore metadata after local copy-back (#8836)
* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback
2026-10-24 02:34:29 +09:00
Chris 5323df7049 fix(protocols): clear clippy failures after T1.5 merge (#8831)
fix(protocols): clear clippy failures after merge
2026-10-23 17:37:26 +09:00
Chris 55b6fbd260 refactor(protocols): move StorageBackend to gateway types (#8825)
* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests
2026-10-23 15:46:28 +09:00
overtrue dd677cc121 Merge main into gateway/integration
Picks up main through 44a73b0be. Conflicts: rustfs/src/error.rs keeps the rustfs-s3-types ApiError conversion and drops the helper main's #8816 replaced; scripts/check_s3s_footprint.sh keeps the integration script, whose baseline lives in .config (main only retuned a constant the integration script does not have). Main's new tests use the rustfs-s3-types error code in admin replication and reach s3s::BodySizeLimitExceeded through the app's test-only s3 facade, so the s3s file count stays at 118; s3_error_lines tightens to 302. The test-count baseline is regenerated; the e2e selections are unchanged.
2026-10-23 08:40:02 +08:00
overtrue 0c2f327974 Merge gateway/integration into the main sync
Brings in #8812 and #8818 from gateway/integration. #8812 predates main's heal error classifier (#8778); its two retryable variants (TooManyOpenFiles, OutdatedXLMeta) move into the classifier's transient arms for storage and disk errors, and its test is kept.
2026-10-23 07:22:20 +08:00
overtrue df42cf1ce8 test(e2e): use a non-default region for the differential negative control
Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).
2026-10-23 07:21:19 +08:00
Chris d290d46b34 test: reserve nextest capacity for progress probes (#8818) 2026-10-23 06:12:31 +09:00
Chris 0d297ed8cd fix(heal): retry transient metadata conflicts (#8812)
* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline
2026-10-23 01:47:55 +09:00
overtrue b8dffa2f1d Merge main into gateway/integration
Picks up the 20 main commits since 62b6f5f3f. The one conflict is the tier free-version recovery test setup: it keeps the integration-only fixture isolation (background recovery disabled while the fixture starts) and main's mock tier registration (#8804 skips scans with no tier). The e2e selections are unchanged by these commits; the test-count baseline is regenerated.
2026-10-22 20:17:09 +08:00
overtrue 62b6f5f3fc Merge main into gateway/integration
The scheduled sync has stopped on conflicts since 2026-10-19. Resolved by keeping both sides: main's restructured delete-rollback path in disk/local.rs with the integration-only native-publication record, main's per-instance region with the rustfs-s3-types Region, main's new lifecycle, rebalance and admin code ported to the persisted lifecycle types and rustfs-s3-types errors, both sides' new scanner tests, and the union of the access_consumer imports. The e2e smoke, full (darwin) and repl-nightly selections and the test-count baseline are regenerated from the merged tree.
2026-10-22 10:14:41 +09:00
Chris be0986115c refactor(s3select): run queries on gateway DTO request shapes (#8794)
* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.
2026-10-22 08:16:48 +09:00
Chris 1bd91c8dad refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)
Part of T1.5 (rustfs/backlog#2746), batch 1.
2026-10-22 06:15:04 +09:00
Chris dcb22bd4d5 refactor(ecstore): move bucket configs onto gateway persisted types (#8709)
* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test
2026-10-21 23:51:23 +09:00
Chris fa17718444 refactor(admin): move error types to rustfs-s3-types (#8440)
* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.
2026-10-21 22:50:27 +09:00
overtrue 99abffc57a refactor(lifecycle): replace s3s domain types with Gateway DTOs
Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.
2026-10-17 07:03:41 +08:00
overtrue fc061c7930 fix(ecstore): retry tier commit intent lock contention
Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.
2026-10-17 01:19:11 +08:00
overtrue d86bf1360f fix(ecstore): recover delete batches with missing versions
Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
2026-10-16 17:32:35 +08:00
overtrue f6097d1df2 fix(ci): restore integration test compilation
Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.
2026-10-16 13:06:24 +08:00
rustfs-cla-assistant[bot] 7e5d32eecf Merge remote-tracking branch 'origin/main' into gateway/integration 2026-10-16 02:09:53 +00:00
overtrue 72da0e21c4 chore: merge main into gateway/integration
Keep the migrated compatibility adapters in their active modules while bringing in upstream fixes. Preserve native publication history on the new create, rename and borrowed-open paths.

Validation:
- `cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(disk::os::native_creation_history_tests::) | test(disk::local::test::borrowed_open_cancellation_retains_started_history_before_execution) | test(disk::local::test::owned_direct_child_open_attempt_consumes_native_history)' --no-fail-fast`: 9/9 passed after restoring all 12 mutation cases at cea78f2590645eb0d303cd7cf56f4dd0d9c00e87; these source blobs are unchanged in this merge.
- `cargo nextest run --locked -p rustfs-s3select-query --lib -E 'test(dispatcher::json_document_schema::tests::)' --no-fail-fast`: 5/5 passed at 5421ba682d1329cc7ebeaac7c6a863da2cea1996, with the same tree.
- `cargo fmt --all --check`: passed.
- `./scripts/check_test_count_ratchet.sh --update`: 52 crates, 22007 tests, 165 ignored.
- Profile baselines use seven actual Darwin listings and seven actual Linux listings.
- Earlier conflict snapshot f5e0cdf0c58aed1ebe53b9243cf52f94832ab049: `cargo nextest run --locked -p rustfs-ecstore -p rustfs --lib -E 'test(disk::os::) | test(bucket::metadata_sys::) | test(server::legacy_compat::) | test(server::layer::tests::request_logging_watchdog) | test(server::http::tests::conditional_object_http::) | test(storage::helper::tests::operation_helper)' --no-fail-fast` passed 218/218 before the borrowed-open fix. This is historical evidence.
2026-10-16 09:49:08 +08:00
Chris 78a0cf321c refactor(replication): move the replication config onto gateway types (#8443)
* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests
2026-10-16 06:19:32 +08:00
overtrue cbe38a120f Merge main into gateway/integration
Merge frozen main acd26afd8b and include
compatibility fixes and measured Darwin/Linux membership corrections
across six files.

Validation on candidate b4c3ab560a:
- All 20 local gates passed.
- Native smoke: 218 passed. Native full: 661 passed.
- Full passed on the same-head failed-only rerun; the initial failure
  remains recorded.
- Gateway remains report-only: 38 passed, 180 failed, under the existing
  exemption.

CI: https://github.com/rustfs/rustfs/actions/runs/38285788334
2026-10-13 03:26:16 +08:00
overtrue db0828aa1c test(ecstore): isolate rollback guard loss fixtures
Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.
2026-10-12 00:42:51 +08:00
overtrue 3ca889b099 fix(ecstore): box nested bucket rollback state
Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.
2026-10-11 20:32:00 +08:00
overtrue e6112c526f fix(ecstore): retain creation fences through rollback
Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.
2026-10-11 16:16:31 +08:00
Chris c5967686f1 fix(ecstore): bound bucket creation stack usage (#8466) 2026-10-11 00:55:29 -07:00
Chris 859de565ad refactor(rpc): serve internode rpc bodies with the owned Body (#8441)
* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries
2026-10-11 12:34:14 +08:00
overtrue 1f7cc593e7 test(ecstore): isolate tier-free recovery fixture startup
The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.
2026-10-09 19:29:50 +08:00
overtrue 1d5a50f094 fix: remove redundant pending closure in disk test
Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.
2026-10-09 14:44:39 +08:00
overtrue 53245be9f4 fix: repair integration checks and IPC test readiness
Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.
2026-10-09 13:09:50 +08:00
rustfs-cla-assistant[bot] c47432e353 Merge remote-tracking branch 'origin/main' into gateway/integration 2026-10-09 02:20:19 +00:00
overtrue f622c1bce0 fix(test): compare legacy body wire parity as raw bytes
Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.
2026-10-09 08:42:46 +08:00
overtrue a99d8f4f89 test(connect): restore one-second drive request budgets
Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.
2026-10-09 08:21:15 +08:00
Chris 4d15b81623 refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)
T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).
2026-10-08 15:59:34 -07:00
Chris 836d30192f feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)
rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).
2026-10-08 15:05:16 -07:00
Chris 860c61ee6d feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)
* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface
2026-10-08 11:58:55 -07:00
Chris f600af4704 test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)
Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740
2026-10-08 10:47:10 -07:00
overtrue 9e747e8475 ci(footprint): count only s3_error! lines still on s3s's macro
s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.
2026-10-09 01:03:21 +08:00
Chris 96fc067765 refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)
* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.
2026-10-08 09:30:37 -07:00
Chris e809b4899e refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)
Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734
2026-10-08 08:43:06 -07:00
Chris 584f03f300 test(ecstore): reach the bucket-config codec through the test facade (#8428) 2026-10-08 07:12:19 -07:00
Chris d52d6fade7 ci: make the s3s footprint baseline exact and lower-only (#8403)
* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734
2026-10-08 06:48:57 -07:00
Chris 3623d6db2d ci: raise the test-count baseline to the integration tree (#8426) 2026-10-08 06:48:40 -07:00
Chris 484f1254ca test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)
Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734
2026-10-08 21:25:56 +08:00
Chris c9c16efbb6 ci: add the per-crate test-count ratchet (#8404)
Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734
2026-10-08 21:25:42 +08:00
Chris f2a59073ca feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)
`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734
2026-10-08 04:11:00 -07:00
Chris c1af94d1d1 refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)
Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734
2026-10-08 02:57:47 -07:00
Chris f84bbfe28f feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411) 2026-10-07 21:01:39 -07:00
rustfs-cla-assistant[bot] 378a2ff7f8 Merge remote-tracking branch 'origin/main' into gateway/integration 2026-10-08 02:18:57 +00:00
rustfs-cla-assistant[bot] b88b53aa6a Merge remote-tracking branch 'origin/main' into gateway/integration 2026-10-07 21:18:17 +00:00
1193 changed files with 76631 additions and 172518 deletions
-1
View File
@@ -18,7 +18,6 @@ These implementation, style, and quality rules apply to all contributions, regar
- Do not include agent, tool, contributor, account, or organization names in branch names.
- Push to the user-requested remote or the repository's configured push remote. Do not hard-code or infer a remote from an account name.
- Check free space before artifact-heavy builds, tests, coverage, or downloads. Re-check before a broad gate when space is tight.
- Before handoff, remove the Cargo target directories and temporary worktrees the task created unless they stay intentionally or hold unmerged work. Identify build output by Cargo's markers (`.rustc_info.json`, `CACHEDIR.TAG`), not by directory name: `target` is also a source directory here.
- Remove only task-owned temporary/build artifacts. Never delete another task's worktree or uncommitted data.
- At handoff, mention disk or cleanup details only when they affected execution or artifacts/worktrees remain intentionally.
- For Docker verification, ensure the tested image contains the intended binary. `buildx build` without `--load` leaves the image in the build cache; use the Makefile's loaded image and rebuild without cache only when stale artifacts are evidenced.
+26 -21
View File
@@ -1,38 +1,43 @@
---
name: rustfs-release-publish
description: Publish a RustFS version through frozen Linux candidate acceptance, desktop completion, manifest approval, byte-preserving publication and post-release follow-up. Use only for an explicit release request.
description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
---
# RustFS Release
Require the exact final SemVer target before version edits or publication. If unspecified, inspect releases/tags and ask with concrete candidates; do not guess the channel or bump. Reuse the user's confirmed target and authorized delivery scope. Independent read-only preflight may continue.
Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
## Invariants
- Cargo and every candidate executable contain the final product version. `<target>-preview.<run_number>` is an immutable candidate identifier, not a public tag or displayed binary version.
- Freeze the exact source, reviewed workflow recipe, toolchain, Console, CLI, dependency lock, previous stable version and validation plan. Newer unrelated main or Console releases do not change those inputs.
- Retain `release/<target>` history when a release branch is used; backport only reviewed fixes. Pin `CANDIDATE_SHA` to its exact CI-validated commit. Never force-push a release branch or move a public tag.
- Build Linux first. Complete field acceptance before macOS/Windows and packaging. Full certification and final acceptance are required before publication; a partial Linux PASS grants no release authority.
- Promote the accepted bytes. Final publication never recompiles binaries or rebuilds packages/images. Infrastructure retries reuse successful artifacts; changed source or inputs require a new candidate and affected acceptance.
- Phase failures block dependents. No latest/default-channel update before versioned delivery. Use the [pipeline runbook](../../../docs/operations/release-pipeline.md) for executable entry points, locks and recovery.
- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
## Read by phase
## Read by Phase
Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
| Phase | Procedure |
|---|---|
| 0 | Clean tree, fetch main/tags, verify access, then complete the mandatory [Console gate](references/console-gate.md). |
| 1–2 | [Prepare the exact source and run branch CI](references/release-candidate.md). Version edits use `rustfs-release-version-bump`. |
| 3 | [Build and verify the Linux candidate](references/preview-release.md). |
| 4–5 | [Run artifact, field, Console and rc acceptance](references/preview-acceptance.md), then complete desktop/package/image/chart validation. |
| Confirmation | Report the complete manifest and obtain the confirmation below. |
| 6 | [Publish the accepted manifest](references/final-release.md). |
| 7 | [Reconcile advisories, milestones, installation references, website and release branch](references/post-release-updates.md). |
| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
| Confirmation | Complete the gate below and end the turn. |
| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
| 7 | [Reconcile Security Advisories, milestones, installation references, and the website; reconcile main and remove the completed release branch](references/post-release-updates.md) only after complete publication. |
## Human confirmation
## Human Confirmation
After all acceptance passes, report the target, source SHA, source CI run, Linux and completion run URLs, final manifest SHA-256, accepted Console/CLI identities, compatibility/performance results and backlog report. Explicitly ask whether to publish this exact manifest and end the turn without dispatching publication.
After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH` and that the current Console asset tag/digest match the preview's recorded build inputs. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, accepted Console asset identity/result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
Only a new affirmative reply to that exact report authorizes Phase 6. An earlier general release request, silence or automatic continuation does not count. Reuse confirmation for retries of that same manifest; do not ask again merely because a destination failed. A changed source, input, manifest or acceptance result invalidates it and requires renewed acceptance/confirmation.
Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, hash, and accepted build inputs; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
## Recovery and reporting
## Recovery
Resume failed infrastructure jobs without rebuilding successful units. Fix product defects on main, backport when needed, and start a new candidate. After a final tag exists, retry the same bytes or use a new patch version; never move the tag. Do not conflate implemented, CI-passed, field-accepted, approved, published and post-release-complete states. Report incomplete work and deviations with evidence; a tag or green build alone is not a completed release.
- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
- If only an external build input changed, record it, create a new preview iteration even when the source SHA is unchanged, and repeat the affected build and artifact acceptance before fresh confirmation. Reuse source checks whose inputs remain valid.
- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting unpublished candidate branches or preview tags requires an explicit decision. Completed release branches follow Phase 7 cleanup; keep release tags.
Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.
@@ -4,7 +4,7 @@ Read during Phase 0, before changing RustFS version files or tags. Follow the pa
### Console release gate
Complete this gate before changing any RustFS version file or creating any RustFS tag. Candidate creation resolves `repos/rustfs/console/releases/latest` once and freezes the exact asset for every build, so a successful Console build alone is insufficient. After freezing, later Console releases do not silently change or invalidate the accepted inputs.
Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
1. Read the latest published Console tag and compare it with Console `main`:
@@ -1,13 +1,22 @@
# Final release publication
# Final Release Publication
Read only after fresh confirmation of the complete candidate report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
Dispatch `release-publish.yml` on main with the successful completion run ID and approved `release-manifest.json` SHA-256. Do not create the tag manually. The workflow revalidates source, bytes and trusted certification, then creates the annotated final tag at `CANDIDATE_SHA`. A recipe change fails closed and requires a new candidate; unrelated main changes do not require a rebuild.
## Phase 6 — Publish the final tag on the validated commit
Require both versioned publication and channel publication jobs to succeed. Verify the final tag's commit and manifest identity; all six platform ZIPs and compatibility aliases; DEB/RPM, checksums, SBOM, provenance and evidence bundle; both multi-architecture image variants in Docker Hub, GHCR and Quay; and the accepted Helm chart in the chart repository. Download a host-compatible final artifact and confirm its digest is exactly the accepted candidate digest and its version/source output agrees.
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH` and the current Console release tag/asset digest still match the accepted preview inputs. If either changed after confirmation, the approval no longer applies; validate the changed inputs with a new preview and obtain fresh confirmation. The final tag goes on the exact commit the preview validated:
For stable releases verify GitHub Latest, R2 download aliases and `latest.json` unless a newer stable release already owns those channels. Prereleases must remain GitHub prereleases and leave stable pointers unchanged. Verify generated release notes compare against the frozen previous stable release, excluding internal previews.
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
git push origin "<target>"
```
On infrastructure failure retry the same workflow/seal; matching destinations are reused. Never clobber an asset, force a tag, rebuild the image or silently accept a changed digest. A source fix after the final tag requires a new patch version. Cross-registry writes are not atomic: report the unfinished destination rather than claiming publication complete.
- CI rebuilds from the same source with a different tag name, so the binary self-reports `<target>`. Other inputs can change: `build.yml` reads Console `releases/latest` during each build. Compare the actual Console tag, URL, and verified digest in every final build job with the accepted preview record, and inspect available build provenance for other relevant changes. A pre-tag check cannot prevent a later input change. On a mismatch or missing evidence, repeat affected acceptance against the final artifacts and report the limitation; failed acceptance leaves release validation incomplete even if publication already succeeded. Never move the final tag to conceal a failure.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed and `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target. Each surviving preview tag must still resolve to its own recorded commit; the accepted preview resolves to `PREVIEW_HASH`. If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Verify a host-compatible final artifact's checksum and `./rustfs --version`; it must report `<target>` and the expected source SHA. This identity check does not replace affected acceptance when build inputs changed.
New candidates create no preview Releases to clean up. For a legacy preview release, preserve its tag and remove its Release only after final delivery and notes are verified. Do not pass `--cleanup-tag`. Complete the [post-release follow-up](post-release-updates.md) separately.
Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.
@@ -24,7 +24,7 @@ Reconcile advisories after every completed final release, including Console-only
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart is packaged and tested before sealing; publication uploads those same bytes only after versioned Docker images are available.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
## rustfs.com top banner
@@ -54,4 +54,4 @@ Use the existing announcement configuration from [website PR #112](https://githu
1. Compare the exact published tag with current main. Audit every release-side commit since their merge base, including inherited backports. An ahead count is a history difference, not proof of missing changes. Use ancestry and patch equivalence, then inspect current code and regression evidence for squashed, adapted, superseded, or reverted changes. Record each commit as present, superseded, release-only, or missing; a positive `git cherry` result alone does not prove a missing fix.
2. Bring missing shared changes to main through a reviewed PR and the required checks before closing the follow-up. Preserve main's later changes; decide version bumps and pinned Console configuration separately. If a history merge is needed, use a merge commit and verify the release commit is an ancestor of main; squash does not establish that ancestry. Do not merge main into the published release branch or move its tags. Report unresolved changes instead of claiming reconciliation is complete.
3. After publication and authorized follow-ups are complete, delete the remote `release/<target>` branch. Immediately before deletion, verify its HEAD still equals the published tag's commit and no open PR targets or originates from it. Preserve a branch with newer work or open PRs and report the blocker. Use a deletion with an expected-old-SHA check to avoid removing a concurrently advanced branch.
4. Verify the remote branch is absent and the final tag and any legacy preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
4. Verify the remote branch is absent and the final and preview tags still resolve to their recorded commits. Remove a local release branch only if it is task-owned, unchecked out, and has no unpreserved work. Record cleanup separately from artifact publication; an explicitly authorized deletion before reconciliation does not make unresolved main changes complete.
@@ -1,10 +1,10 @@
# Candidate Artifact Acceptance
# Preview Artifact Acceptance
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and the rc release frozen in the candidate inputs.
Read after Phase 3 succeeds. Complete every check below before the parent skill's manual confirmation gate. These checks cover the downloaded artifact, embedded Console, and latest rc client.
## Phase 4 — Run the artifact locally, verify the console
Use a unique task-owned scratch directory. Select the versioned archive matching the acceptance host, and verify it against the sealed manifest before extraction. Before desktop completion, use the Linux acceptance environment rather than rebuilding for the local desktop. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Use a unique task-owned scratch directory. Select the versioned archive matching the host OS, architecture, and runtime, download its checksum file, and verify the archive before extraction. Do not substitute a different build when a matching artifact is unavailable; report that acceptance gap. The shell example below assumes the verified executable is at `RUSTFS_BINARY` (`rustfs.exe` on Windows), `SCRATCH` is the task directory, and `S3_PORT` and `CONSOLE_PORT` are distinct available ports. Use equivalent native commands when the shell differs.
Bind both listeners to loopback. Choose task-only credentials, set loopback proxy exclusions for the server/client processes, and record the server PID. A port probe does not reserve a port; retry startup with fresh ports on a bind conflict rather than connecting to an existing service.
@@ -24,16 +24,16 @@ PREVIEW_SERVER_PID=$!
Checks (all must pass):
- `"$RUSTFS_BINARY" --version` reports the exact final `<target>` and full `CANDIDATE_SHA`. A preview suffix, another version or another source is a failure.
- `"$RUSTFS_BINARY" --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
- Wait with a bounded readiness deadline while checking the recorded server process remains alive; `curl --noproxy '*' -fsS "$S3_ENDPOINT/health/ready"` must return ready from this process's listener.
- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
- Open `CONSOLE_ENDPOINT` in the browser and log in with the task credentials. The dashboard must render without JS console errors; create a bucket, upload a file, download it back (byte-identical), and delete the object and bucket. Keep the server running for Phase 5.
## Phase 5 — Validate with the pinned rc client
## Phase 5 — Validate with the latest rc client
`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
- Read the CLI release and asset digests from `inputs.json`; verify the local client's version or download a host-compatible asset from that exact release into the task directory. If the host is a desktop, resolve its asset within the same pinned CLI release and record the digest; do not independently follow `latest`. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Resolve the latest release with `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; verify the local client's version or download and verify a host-compatible client into the task directory. Set `RC_BIN` to that executable and record its version/digest. Avoid changing the user's installed client for this check.
- Inspect the client's current help for an isolated configuration option and use it when supported. Otherwise choose a unique `PREVIEW_ALIAS` and confirm it is absent before creating it; never overwrite an existing alias. Use task-owned bucket/user names and input/output files. The names below are safe only inside the newly started isolated server.
- Point the client at `S3_ENDPOINT` with loopback proxy exclusions and run the command matrix, recording PASS/FAIL per command:
@@ -60,9 +60,3 @@ cmp <local-file> "$SCRATCH/out"
```
- Any FAIL blocks acceptance. On success or failure, stop only the recorded task process and remove only task-created aliases/data after retaining the needed failure evidence. Do not erase an existing client configuration or unrelated server data.
## Field acceptance and desktop completion
Before accepting the Linux manifest, record Scanner/Heal, multi-pool, memory, major features, high-concurrency 503, performance and previous-version integrity/upgrade results in backlog. Preserve the release-specific agreed limits and baseline; a PASS label without artifact identities or test results does not satisfy acceptance.
Dispatch `release-complete.yml` only after that report passes, using the exact Linux manifest digest and backlog URL. Require native macOS/Windows checks, package extraction, image startup/scans, chart tests and complete certification to pass. Complete any desktop Console/client checks applicable to the release against these same archives. Record the completion run and final manifest SHA-256 before entering the parent confirmation gate. Partial Linux acceptance does not authorize final publication.
@@ -1,9 +1,25 @@
# Linux candidate verification
# Preview Release Verification
Read for Phase 3 after the source passed exact-SHA CI. The [parent invariants](../SKILL.md) apply; commands and recovery live in the [pipeline runbook](../../../../docs/operations/release-pipeline.md).
Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
Dispatch `release-candidate.yml` on main with the confirmed final version and `CANDIDATE_SHA`. Record the run ID, frozen `inputs.json`, every Linux archive digest and `linux-manifest.json` SHA-256. Require the four Linux builds, native startup/S3 checks, previous-release upgrade matrix and Linux seal to succeed. Missing, skipped, cancelled or failed required work is incomplete acceptance.
## Preview channels
Verify the frozen Console identity equals the one accepted in Phase 0. Every matrix job consumes that same digest and exact toolchain. The binary must report the final product version and source commit; its candidate ID is separate. No public tag, Release, registry tag, R2 release path, Helm repository or latest pointer should change during this stage.
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
A green Linux run is ready for field acceptance, not publication. The partial certification must have `publish_allowed: false`. Retain the exact Linux manifest for the next stage; a new source/build-input candidate must not inherit the old approval. A newer unrelated main commit does not invalidate this evidence.
## Phase 3 — CI and preview Release verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Record the inputs actually consumed by each build job: source SHA and the embedded Console release tag, asset URL, and verified SHA256 from the download logs. The workflow reads Console `releases/latest` separately for each build; confirm the matrix used the Console asset accepted in Phase 0. A missing identity or differing asset blocks acceptance until resolved. Retain available toolchain and build provenance with the artifact checksums; the same source SHA alone does not establish equivalent builds.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
## Release notes
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
@@ -16,13 +16,22 @@ Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a candidate build is dispatched.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
## Phase 2 — Pin and validate the release source
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. A legacy preview can anchor source selection, but its tag and old artifacts do not constitute a candidate in the new protocol; freeze and verify a new candidate after the release recipe is aligned.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not dispatch a candidate with failing or incomplete CI.
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
- Set `CANDIDATE_SHA` to the CI-validated release-branch commit. Record the branch, exact commit and CI run URL. The new candidate protocol starts from a commit whose release recipe matches the reviewed main dispatcher; backport the recipe before freezing an older release branch if needed.
- Dispatch the Linux candidate as described in [candidate verification](preview-release.md). Do not create a preview or final tag here. The final executable already contains `<target>`; publication later promotes those same bytes.
- After a source or build-input change, run the affected source checks and create a new candidate. Reuse the frozen candidate after an infrastructure failure. Do not restart acceptance for unrelated main advances.
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
@@ -4,18 +4,18 @@ description: Prepare an exact RustFS Cargo version or align installation referen
---
# Release Version Files
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those identify candidates, not product versions or new public tags. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
| Stage | Files and constraints |
|---|---|
| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Candidate preparation retains the previous published installation references; follow-up commits never replace validated tags or sealed artifacts.
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
Packaging policy:
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>`; both equal the final product SemVer, including prerelease suffixes. Helm CI uses the frozen candidate version before sealing, so no early installation chart bump is required. Previously published chart versions remain unchanged.
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
- Changing these mappings requires explicit confirmation.
-16
View File
@@ -1,16 +0,0 @@
{
"schema_version": 1,
"rollout": "enforced",
"core_packages": ["rustfs", "rustfs-ecstore", "rustfs-filemeta", "rustfs-iam", "rustfs-policy"],
"connect_packages": [
"rustfs-credentials", "rustfs-config", "rustfs-common", "rustfs-crypto",
"rustfs-iam", "rustfs-policy", "rustfs-kms", "rustfs-license", "rustfs-security-governance",
"rustfs-utils", "rustfs-tls-runtime", "rustfs-trusted-proxies", "rustfs-filemeta",
"rustfs-lock", "rustfs-obs", "rustfs-signer", "rustfs-io-metrics", "rustfs-s3-ops"
],
"full_paths": [
"Cargo.toml", "Cargo.lock", "*/Cargo.toml", "rust-toolchain*", "rustfmt.toml",
"clippy.toml", ".cargo/*", ".config/*", ".github/*", "Makefile",
"scripts/*", "build.rs", "*.proto"
]
}
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-linux=348fa5eed9ae58962c06b33283ff4f15122cff97bce9b4eb1e3ffcb227264b0b
sha256-darwin=348fa5eed9ae58962c06b33283ff4f15122cff97bce9b4eb1e3ffcb227264b0b
sha256-linux=eb33067d90ee66c0a97e8f3658e46c152e47f897d2fbfdc6cb5ad93cfa55414b
sha256-darwin=eb33067d90ee66c0a97e8f3658e46c152e47f897d2fbfdc6cb5ad93cfa55414b
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=9901a4950099b94d3d2618524d540f1ed4bf6023044903886f98b6c4bf5f30fb
sha256-linux=09fd5f48f5556c9fc576357222dccd7ebb0b90b790798f1834db2ec5d67b4952
sha256-darwin=4fb92e88763982eabafd74d05eb3d269387624a447e427a8c6f84914e5b03bfe
sha256-linux=bd0db7e846ce24f056a097655db8fe2265201913260d0e16feb38d99dcb954a7
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=e84e04208f19dc0596b10d4d1b799ce43d6cc431415902f9c5458aa8232bc2d8
sha256-linux=1f9c7add72c44d1c12ef4ac050bb7917431be657c0c90b29f70c3df56f8bbae3
sha256-darwin=3814cdde8cacf4187e18188d342ed70931cc32e268e42e5bb438d565bb4d0f81
sha256-linux=663025a80273579f95ae3f49af02c0afcbfce94df09701c360a13e607b7637db
+1 -1
View File
@@ -1 +1 @@
sha256=e4ee3898b840a8374cb02c64342cc15f8d7ecd1becaf4b4a2a4657290f9e7ff1
sha256=4eac269f9a59b2f9be4e7a87ea6aa38454f5ebe10ede35c9b8d4218137d2cd35
+1 -1
View File
@@ -1 +1 @@
sha256=5bdd4c2eadbc41625ae3184fff34062e000d328108436259b2b7c4fe2531ecaf
sha256=879f421f1115d44f930e229510a2e533cfd5ebd4effe38730b82ef61f4fd7a86
+5 -1
View File
@@ -70,6 +70,11 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
@echo "📦 Checking s3s footprint ratchet..."
./scripts/check_s3s_footprint.sh
.PHONY: test-count-ratchet-check
test-count-ratchet-check: ## Check no workspace crate lost tests or gained #[ignore] against the committed baseline
@echo "🧪 Checking per-crate test-count ratchet..."
./scripts/check_test_count_ratchet.sh
.PHONY: fips-wording-check
fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
@echo "📣 Checking cryptographic capability wording guard..."
@@ -90,7 +95,6 @@ test-wiring-check: ## Check tests stay registered and selected by their intended
@echo "🧪 Checking test wiring..."
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py
$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --check-workflow
$(RUSTFS_PYTHON_BIN) ./scripts/validation_registry.py check
.PHONY: log-analyzer-rules-check
log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
+2 -2
View File
@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY: pre-commit
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY: pre-pr
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY: dev-check
+2 -18
View File
@@ -22,21 +22,12 @@ TEST_THREADS ?= 1
# differ from CI and .config/nextest.toml test-groups will NOT apply.
.PHONY: script-tests
.PHONY: release-package-tests
release-package-tests: ## Run release package integration on Linux with fpm, dpkg and rpm installed
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_packages.py
.PHONY: release-image-tests
release-image-tests: ## Run OCI integration with Docker, Go and skopeo; set RELEASE_IMAGE_VARIANT and RELEASE_IMAGE_OUTPUT
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_images.py
script-tests: ## Run shell script tests
@echo "Running script tests..."
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
./scripts/test_build_rustfs_options.sh
./scripts/test_docker_runtime_timezone.sh
./scripts/test_entrypoint_credentials.sh
./scripts/test_entrypoint_directories.sh
./scripts/test_internode_grpc_ab_bench.sh
./scripts/test_object_batch_bench_enhanced.sh
./scripts/test_package_service_scripts.sh
@@ -60,29 +51,22 @@ script-tests: ## Run shell script tests
./scripts/test_python_bin.sh
$(RUSTFS_PYTHON_BIN) ./scripts/test_check_doc_paths.py
./scripts/check_embedded_secrets.sh --self-test
./scripts/check_s3s_footprint.sh --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_download_release_asset.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/release -p "test_*.py"
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_connect_release_boundary.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_test_artifact.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_test_shards.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_merge_queue.py
$(RUSTFS_PYTHON_BIN) -m unittest discover -s scripts/ci -p 'test_*.py'
$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_scheduled_validation_freshness.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_security_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_nightly_candidate.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_repro_rejoin_readiness_docker.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_validation_registry.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_validation_runner.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_retain_validation_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
+27 -32
View File
@@ -110,7 +110,7 @@ threads-required = "num-test-threads"
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_object_registration_completes_while_other_bucket_backup_waits)'
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.default.overrides]]
@@ -234,7 +234,7 @@ test-group = 'embedded-test-ports'
# processes cannot exhaust those budgets. Keep deadlines and assertions intact.
# The short HTTP fixtures are tracked in rustfs/backlog#2714.
[[profile.default.overrides]]
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|cleanup_accepts_missing_object_and_still_deletes_the_bucket|cleanup_diagnostics_identify_each_disconnected_transport_stage|cleanup_diagnostics_identify_each_failed_http_stage_without_private_context|cleanup_diagnostics_identify_timeouts_without_resetting_the_shared_deadline|oversized_response_is_rejected_without_buffering_it)$/)))'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
threads-required = "num-test-threads"
# This cancellation probe hashes the serving executable before acquiring its
@@ -356,6 +356,12 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-ecstore) & test(=store::init::tests::dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member)'
threads-required = "num-test-threads"
# These short-deadline real-disk progress probes need the test run's capacity
# for scanner and object-lock handoffs; keep their assertions and deadlines.
[[profile.default.overrides]]
filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
threads-required = "num-test-threads"
# ---------------------------------------------------------------------------
# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
# ---------------------------------------------------------------------------
@@ -410,7 +416,7 @@ threads-required = "num-test-threads"
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_object_registration_completes_while_other_bucket_backup_waits)'
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.ci.overrides]]
@@ -424,6 +430,12 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
threads-required = "num-test-threads"
# Keep short-deadline real-disk progress probes from competing with unrelated
# tests for scanner and object-lock capacity. Assertions and deadlines stay on.
[[profile.ci.overrides]]
filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
threads-required = "num-test-threads"
# Emit this expensive fixture's stage timings in CI and use tmpfs when safe.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(=store::init::tests::decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page)'
@@ -548,7 +560,7 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
test-group = 'embedded-test-ports'
[[profile.ci.overrides]]
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|cleanup_accepts_missing_object_and_still_deletes_the_bucket|cleanup_diagnostics_identify_each_disconnected_transport_stage|cleanup_diagnostics_identify_each_failed_http_stage_without_private_context|cleanup_diagnostics_identify_timeouts_without_resetting_the_shared_deadline|oversized_response_is_rejected_without_buffering_it)$/)))'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
threads-required = "num-test-threads"
# Preserve the cancellation probe's startup budget and assertions in CI too.
@@ -698,14 +710,23 @@ test-group = 'ecstore-serial-flaky'
# the PR-lane e2e coverage for scanner usage snapshots consumed by quota and
# admin surfaces. It uses the same single-node, random-port, isolated-temp-dir
# fixture as the existing smoke modules.
#
# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
# starts two single-node servers from the same binary, one per
# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
# SDK script covering every operation of `impl S3 for FS` against both. Its
# comparator unit tests need no server. It meets the admission criteria
# unchanged; the module clause admits a new differential test without another
# filter edit.
[profile.e2e-smoke]
default-filter = """
package(e2e_test) & (
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|multipart_upload_id|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat|data_usage)_test::|^fake_s3_target::/)
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat|data_usage)_test::|^fake_s3_target::/)
| test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
| test(/^reliant::lifecycle::/)
| test(/^conditional_delete_objects_test::/)
| test(/^reliant::tiering::/)
| test(/^dual_stack_differential::/)
| test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
| test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
)
@@ -827,7 +848,7 @@ slow-timeout = { period = "120s", terminate-after = 12, grace-period = "10s" }
# ---------------------------------------------------------------------------
# e2e-distributed profile — 4-node 4-disk Actions suite
# ---------------------------------------------------------------------------
# Nightly and manual-dispatch lane owned by
# Storage-sensitive PR / nightly / dispatch lane owned by
# .github/workflows/e2e-distributed.yml.
# Each case starts four rustfs processes (and for site replication, two
# clusters). Upgrade cases also require RUSTFS_UPGRADE_SOURCE_BINARY.
@@ -960,12 +981,6 @@ test-group = 'e2e-inline-boundaries'
filter = 'package(e2e_test) & test(/^heal_erasure_disk_rebuild_test::/)'
test-group = 'e2e-cluster-nightly'
# The multi-pool crash fixture seeds three nodes and 36 drives. Reserve this
# nextest run's workers so other multi-node tests do not starve its deadline.
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(=heal_erasure_disk_rebuild_test::tests::test_cluster_root_heal_recovers_ec84_shards_across_multi_pool_after_background_target_crash)'
threads-required = "num-test-threads"
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
test-group = 'e2e-vault'
@@ -975,23 +990,3 @@ test-group = 'e2e-vault'
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
threads-required = "num-test-threads"
# The mc fixture uploads hundreds of objects concurrently before measuring listing.
# Reserve the runner so other server fixtures do not starve its disk operations.
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(=mc_mirror_small_bucket_test::test_mc_mirror_small_bucket_completes_without_list_timeout)'
threads-required = "num-test-threads"
# Fixed, small contract for every executable server change. These existing
# scenarios exercise write/read, anonymous denial and retained bytes on restart.
[profile.core-contract]
default-filter = '''
package(e2e_test) & test(/^(anonymous_access_test::test_anonymous_access_denied_when_restrict_public_buckets_enabled|head_object_consistency_test::head_object_consistency_after_write_and_multipart_and_presigned_head|storage_class_capability_test::tests::historical_label_only_metadata_is_standard_without_hiding_a_real_transition_tier)$/)
'''
retries = 0
fail-fast = false
test-threads = 1
slow-timeout = { period = "60s", terminate-after = 3 }
[profile.core-contract.junit]
path = "junit.xml"
-135
View File
@@ -1,135 +0,0 @@
{
"schema_version": 1,
"plan_id": "rustfs-release-validation",
"description": "Certification for immutable candidates. All shipped platforms require native identity and startup; stable and prerelease require the pinned upgrade matrix. Linux-only acceptance never authorizes publication.",
"release_types": [
"preview",
"prerelease",
"stable"
],
"targets": [
{
"os": "linux",
"arch": "x86_64",
"abi": "musl"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "musl"
},
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
},
{
"os": "linux",
"arch": "aarch64",
"abi": "gnu"
},
{
"os": "macos",
"arch": "aarch64",
"abi": "none"
},
{
"os": "windows",
"arch": "x86_64",
"abi": "msvc"
}
],
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
"features": [
"pyroscope"
],
"trusted_workflows": [
{
"repository": "rustfs/rustfs",
"workflow_id": "release-candidate.yml"
},
{
"repository": "rustfs/rustfs",
"workflow_id": "release-complete.yml"
}
],
"trusted_workflows_basis": "Receipts are produced by the reviewed main workflows and fetched by exact run ID with GitHub artifact digest verification. Candidate inputs freeze the harness commit separately from product source.",
"lanes": [
{
"lane": "identity",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 1800,
"environment_capabilities": [
"process-exec"
],
"cases": [
"artifact-identity",
"startup"
],
"applies_to": {
"release_types": [
"preview",
"prerelease",
"stable"
],
"platforms": "all"
},
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
},
{
"lane": "upgrade",
"harness_repository": "rustfs/rustfs",
"harness_sha": "candidate.workflow_sha",
"harness_basis": "The exact workflow commit is frozen in candidate inputs; no mutable branch is accepted as a harness identity.",
"timeout_seconds": 5400,
"environment_capabilities": [
"process-exec",
"previous-release-artifact"
],
"cases": [
"direct-upgrade",
"mixed-version-upgrade",
"bucket-config-upgrade",
"bucket-config-rollback",
"multipart-layout-upgrade",
"multipart-layout-baseline"
],
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
"applies_to": {
"release_types": [
"prerelease",
"stable"
],
"platforms": [
{
"os": "linux",
"arch": "x86_64",
"abi": "gnu"
}
]
},
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
}
],
"upgrade_sources": {
"basis": "Exactly the two previous-release sources pinned in .github/workflows/e2e-upgrade.yml are exercised today (source.version / source.asset / source.sha256). No other N-1/N-2 compatibility is claimed, and these digests are not inferred from tags.",
"sources": [
{
"version": "1.0.1",
"role": "N-1",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.1.zip",
"artifact_sha256": "9742e183cb0dfb0d48fa909797af572fe47b7f827569036baa4156e0d3958ee5"
},
{
"version": "1.0.0-rc.5",
"role": "N-2",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip",
"artifact_sha256": "3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d"
}
],
"rollback_supported": false,
"rollback_basis": "Only the bucket-config-rollback case (rollback_to_previous_release_reads_current_bucket_metadata) is exercised. Full data-plane rollback is not certified and is never a default claim."
}
}
+37
View File
@@ -0,0 +1,37 @@
# Files that may still reference s3s (`s3s::` or `s3s_sigv4`) when Phase 1 of
# the gateway migration ends (rustfs/backlog#2734 task T0.6, rustfs/backlog#2741).
# Consumed only by `scripts/check_s3s_footprint.sh --mode allowlist`.
#
# One entry per line: a repository-relative file path or a git `:(glob)`
# pathspec. `**` matches across directories, `*` stays within one path
# component, and a directory entry matches everything under it. Blank lines
# and lines starting with `#` are ignored; an entry may not contain whitespace,
# so a note belongs on its own comment line.
#
# Every Rust file listed by `git grep -l '\bs3s\(::\|_sigv4\)' -- '*.rs'` that
# no entry matches fails the allowlist mode. Task T1.9 switches CI to that
# mode; until then `--dry-run` prints the files still outside this list, which
# is the Phase 1 tracking list.
#
# Never add crates/** other than crates/e2e_test: the engine crates leave s3s
# entirely, and the script rejects such an entry. Only the s3s-facing edge of
# the rustfs binary and the e2e oracle keep it.
# rustfs binary: the s3s-facing edge that survives Phase 1
rustfs/src/storage/ecfs.rs
rustfs/src/storage/s3_compat/**
rustfs/src/storage/access.rs
rustfs/src/auth.rs
rustfs/src/error.rs
rustfs/src/storage_api.rs
rustfs/src/app/storage_api.rs
rustfs/src/admin/router.rs
rustfs/src/app/metadata_route.rs
rustfs/src/server/http.rs
rustfs/src/server/layer.rs
rustfs/src/server/hybrid.rs
rustfs/src/protocols/client.rs
# e2e oracle: the test target that verifies S3 behaviour against s3s
crates/e2e_test/src/fake_s3_target/**
crates/e2e_test/src/**
+17
View File
@@ -0,0 +1,17 @@
# Exact baseline for the direct s3s footprint, consumed only by
# scripts/check_s3s_footprint.sh (rustfs/backlog#1677; rustfs/backlog#2734 T0.4).
#
# Each value is the exact measurement of the tree, not a ceiling. CI fails when a
# measured counter differs from its line here in either direction:
# - a larger measurement means new code widened the s3s surface: route it
# through the gateway abstractions instead; never raise a value here;
# - a smaller measurement means the PR shrank the surface: lower the matching
# value to the measured number in the same PR.
#
# files Rust files referencing s3s paths, repo-wide minus crates/e2e_test
# s3_error_lines lines holding s3_error!, same scope, minus those the script
# header's rule clears (rustfs_s3_types's macro); fails closed
# ecstore_files Rust files referencing s3s paths under crates/ecstore/src
files=65
s3_error_lines=289
ecstore_files=0
+65
View File
@@ -0,0 +1,65 @@
# Per-crate test-count baseline consumed by scripts/check_test_count_ratchet.sh
# (rustfs/backlog#2734 section 1 rule 3; T0.3 in rustfs/backlog#2738).
#
# One row per workspace member: <package>, tests, ignored, tab-separated.
# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
# attribute lines under the package directory; ignored counts #[ignore] and
# #[ignore = "..."]. Both columns are exact: CI fails when a crate's tests fall
# below or rise above its row, or when its ignored count changes. Raise a row in
# the PR that adds tests; lowering one is a reviewed statement that tests were
# intentionally removed. Regenerate with:
# scripts/check_test_count_ratchet.sh --update
e2e_test 993 20
rustfs 6078 64
rustfs-audit 70 0
rustfs-checksums 26 0
rustfs-common 36 0
rustfs-concurrency 54 0
rustfs-config 43 0
rustfs-connect-symbol-catalog 6 0
rustfs-credentials 24 0
rustfs-crypto 67 0
rustfs-data-usage 56 0
rustfs-ecstore 7415 2
rustfs-extension-schema 23 0
rustfs-filemeta 353 0
rustfs-gateway-adapter 11 0
rustfs-heal 1033 3
rustfs-heal-contracts 8 0
rustfs-iam 401 0
rustfs-io-core 49 0
rustfs-io-metrics 154 0
rustfs-keystone 29 0
rustfs-kms 715 25
rustfs-license 2 0
rustfs-lifecycle 155 0
rustfs-lock 143 0
rustfs-log-analyzer 90 0
rustfs-madmin 210 0
rustfs-notify 139 0
rustfs-object-capacity 92 0
rustfs-object-data-cache 140 0
rustfs-obs 385 0
rustfs-policy 734 0
rustfs-protocols 748 1
rustfs-protos 73 0
rustfs-replication 209 0
rustfs-rio 206 0
rustfs-rio-v2 34 6
rustfs-s3-client 69 0
rustfs-s3-ops 8 0
rustfs-s3-types 103 0
rustfs-s3select-api 207 0
rustfs-s3select-query 189 2
rustfs-scanner 1261 20
rustfs-scanner-metrics 90 0
rustfs-security-governance 20 0
rustfs-signer 31 0
rustfs-storage-api 34 0
rustfs-targets 603 26
rustfs-test-utils 1 0
rustfs-tls-runtime 43 0
rustfs-trusted-proxies 68 0
rustfs-utils 247 0
rustfs-zip 30 0
zstd 11 0
+3 -33
View File
@@ -22,9 +22,9 @@ The following files in the project root are essential for Docker operations:
| File | Description | Usage |
| :--- | :--- | :--- |
| **`docker-buildx.sh`** | **Multi-Arch Build Script**<br>Automates building and pushing Docker images for `amd64` and `arm64`. Supports release and dev channels. | `./docker-buildx.sh --push` |
| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads verified RustFS and `rc` binaries from GitHub Releases and bundles `/etc/rustfs/bootstrap`. | `docker build -t rustfs:latest .` |
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Includes the bundled bootstrap configuration. | `docker build -f Dockerfile.glibc .` |
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code and bundles `rc` and the default bootstrap configuration. | `docker build -f Dockerfile.source .` |
| **`Dockerfile`** | **Production Image (Alpine)**<br>Lightweight image using musl libc. Downloads pre-built binaries from GitHub Releases. | `docker build -t rustfs:latest .` |
| **`Dockerfile.glibc`** | **Production Image (Ubuntu)**<br>Standard image using glibc. Useful if you need specific dynamic libraries. | `docker build -f Dockerfile.glibc .` |
| **`Dockerfile.source`** | **Development Image**<br>Builds RustFS from source code. Includes build tools. Ideal for local development and CI. | `docker build -f Dockerfile.source .` |
### Docker Compose Configurations
@@ -110,36 +110,6 @@ docker compose up -d
---
## Docker Compose environment values
Use map syntax when quoting environment values in YAML:
```yaml
services:
rustfs:
image: rustfs/rustfs:latest
ports:
- "9000:9000"
- "9001:9001"
volumes:
- ./data:/data
- ./logs:/var/log/rustfs
environment:
RUSTFS_ADDRESS: ":9000"
RUSTFS_CONSOLE_ADDRESS: ":9001"
RUSTFS_CONSOLE_ENABLE: "true"
RUSTFS_OBS_LOGGER_LEVEL: error
RUSTFS_OBS_LOG_DIRECTORY: "/var/log/rustfs/"
RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:?Set RUSTFS_ACCESS_KEY}"
RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:?Set RUSTFS_SECRET_KEY}"
```
Create the host directories and grant the container user write access before starting this example; official images run as UID/GID `10001:10001`. Set both credential variables to your own values, then run `docker compose config` to inspect the resolved configuration locally before `docker compose up -d`. The resolved output includes credentials, so do not share it publicly.
With list syntax, use `- RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs/` or quote the whole item as `- "RUSTFS_OBS_LOG_DIRECTORY=/var/log/rustfs/"`. Writing `- RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"` passes the inner quotes literally, making this a relative path beginning with `"`, which can fail with `mkdir: cannot create directory '"': Permission denied`. This also applies to addresses and credentials: fix every affected list item, rather than only the log directory. The entrypoint preserves configured values, including literal quotes in valid paths and secrets.
---
## 🔧 Common Operations
### Cleaning Up
@@ -28,15 +28,6 @@ processors:
check_interval: 1s
limit_mib: 1024
spike_limit_mib: 256
transform/metrics:
error_mode: ignore
metric_statements:
# RustFS process metrics already expose process_pid as a metric label.
# Remove only the duplicate datapoint label and keep the shared resource
# process.pid available to sibling metrics and lock histograms.
- context: datapoint
statements:
- delete_key(attributes, "process_pid") where IsMatch(metric.name, "^rustfs_system_process_")
transform/logs:
error_mode: ignore
log_statements:
@@ -121,7 +112,7 @@ service:
exporters: [ otlp/tempo, otlp/jaeger ]
metrics:
receivers: [ otlp ]
processors: [ transform/metrics, memory_limiter, batch ]
processors: [ batch ]
exporters: [ prometheus ]
logs:
receivers: [ otlp ]
-40
View File
@@ -1,40 +0,0 @@
# syntax=docker/dockerfile:1.6
# Copyright 2026 RustFS Team
# Licensed under the Apache License, Version 2.0.
# A compatible native runtime with Python 3.11+, getent and stat is required by
# the isolated fixture's staging/provenance checks. This is a test image.
ARG RUNTIME_BASE=python:3.13-slim-trixie
FROM ${RUNTIME_BASE}
ARG SOURCE_REVISION
ARG BINARY_SHA256
ARG BUILD_PROFILE=dev
COPY rustfs /usr/bin/rustfs
RUN python3 - "$SOURCE_REVISION" "$BINARY_SHA256" "$BUILD_PROFILE" <<'PY'
import hashlib
import os
import re
import shutil
import sys
source, digest, profile = sys.argv[1:]
if not re.fullmatch(r"[0-9a-f]{40}", source):
raise SystemExit("invalid source revision")
if not re.fullmatch(r"[0-9a-f]{64}", digest):
raise SystemExit("invalid binary digest")
if profile not in ("dev", "release"):
raise SystemExit("invalid build profile")
with open("/usr/bin/rustfs", "rb") as binary:
if hashlib.file_digest(binary, "sha256").hexdigest() != digest:
raise SystemExit("binary digest mismatch")
for tool in ("getent", "stat"):
if shutil.which(tool) is None:
raise SystemExit(f"missing runtime tool: {tool}")
os.chmod("/usr/bin/rustfs", 0o755)
PY
LABEL io.rustfs.repro.source_revision="${SOURCE_REVISION}" \
io.rustfs.repro.binary_sha256="${BINARY_SHA256}" \
io.rustfs.repro.build_profile="${BUILD_PROFILE}"
USER 10001:10001
ENTRYPOINT ["/usr/bin/rustfs"]
+9 -19
View File
@@ -13,7 +13,7 @@
# limitations under the License.
name: Quick Checks
description: Run the shared RustFS quality checks.
description: Run the shared compile-free RustFS quality checks.
runs:
using: composite
@@ -24,30 +24,19 @@ runs:
tool: |
ripgrep@15.2.0
shellcheck@0.11.0
cargo-nextest@0.9.138
- name: Install actionlint
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/../setup/download_release_asset.py
run: |
actionlint_dir="$(mktemp -d "${RUNNER_TEMP}/actionlint.XXXXXX")"
python3 "$ASSET_DOWNLOADER" --repository rhysd/actionlint --tag v1.7.12 --asset actionlint_1.7.12_linux_amd64.tar.gz --output "$actionlint_dir/actionlint.tar.gz" --sha256 8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8
curl --fail --location --silent --show-error \
--output "$actionlint_dir/actionlint.tar.gz" \
https://github.com/rhysd/actionlint/releases/download/v1.7.12/actionlint_1.7.12_linux_amd64.tar.gz
echo "8aca8db96f1b94770f1b0d72b6dddcb1ebb8123cb3712530b08cc387b349a3d8 $actionlint_dir/actionlint.tar.gz" | sha256sum --check --status
tar -xzf "$actionlint_dir/actionlint.tar.gz" -C "$actionlint_dir" actionlint
rm "$actionlint_dir/actionlint.tar.gz"
echo "$actionlint_dir" >> "$GITHUB_PATH"
- name: Install Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install workflow parser
shell: bash
run: python3 -m pip install --disable-pip-version-check PyYAML==6.0.3
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
with:
@@ -97,6 +86,10 @@ runs:
shell: bash
run: ./scripts/check_s3s_footprint.sh
- name: Check per-crate test-count ratchet
shell: bash
run: ./scripts/check_test_count_ratchet.sh
- name: Check cryptographic capability wording
shell: bash
run: ./scripts/check_fips_wording.sh
@@ -115,10 +108,7 @@ runs:
- name: Check test wiring
shell: bash
run: |
python3 ./scripts/check_test_wiring.py
python3 ./scripts/ci_gate.py --check-workflow
python3 ./scripts/validation_registry.py check
run: python3 ./scripts/check_test_wiring.py
- name: Check no planning docs committed
shell: bash
-107
View File
@@ -1,107 +0,0 @@
name: Runner capability and build pilot
description: Inspect an existing runner and measure isolated fixed-source cold/warm builds
inputs:
workload:
description: probe, server or workspace-tests
required: true
source-sha:
description: Exact merged main commit for builds
default: ""
toolchain:
description: Exact Rust version for builds
default: ""
scratch-root:
description: Existing absolute scratch root
default: ""
build-jobs:
description: Cargo concurrency
default: "2"
local-ssd:
description: Require operator storage attestation
default: "false"
runs:
using: composite
steps:
- name: Validate pilot input and inspect existing capabilities
shell: bash
env:
PILOT_WORKLOAD: ${{ inputs.workload }}
PILOT_SHA: ${{ inputs.source-sha }}
PILOT_TOOLCHAIN: ${{ inputs.toolchain }}
PILOT_SCRATCH: ${{ inputs.scratch-root }}
PILOT_JOBS: ${{ inputs.build-jobs }}
PILOT_SSD: ${{ inputs.local-ssd }}
run: |
[[ "$PILOT_WORKLOAD" =~ ^(probe|server|workspace-tests)$ ]]
[[ "$PILOT_JOBS" =~ ^([1-9]|1[0-6])$ ]]
if [[ "$PILOT_WORKLOAD" != probe ]]; then
[[ "$PILOT_SHA" =~ ^[a-f0-9]{40}$ ]]
[[ "$PILOT_TOOLCHAIN" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
fi
scratch="${PILOT_SCRATCH:-$RUNNER_TEMP}"
[[ "$scratch" == /* && -d "$scratch" ]]
minimum_disk=1
minimum_memory=0
if [[ "$PILOT_WORKLOAD" == server ]]; then minimum_disk=24; minimum_memory=8; fi
if [[ "$PILOT_WORKLOAD" == workspace-tests ]]; then minimum_disk=90; minimum_memory=16; fi
extra=()
if [[ "$PILOT_SSD" == true ]]; then extra+=(--local-ssd-attestation /etc/rustfs-runner/storage.json); fi
python3 scripts/ci/runner_pilot.py --root "$scratch" \
--output "$RUNNER_TEMP/runner-pilot-evidence" \
--min-free-gib "$minimum_disk" --min-memory-gib "$minimum_memory" --architecture x86_64 "${extra[@]}"
- name: Checkout the trusted baseline history
if: inputs.workload != 'probe'
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: main
path: .runner-pilot-source
fetch-depth: 0
persist-credentials: false
- name: Bind the build to a merged main commit
if: inputs.workload != 'probe'
shell: bash
env:
PILOT_SHA: ${{ inputs.source-sha }}
run: |
git -C .runner-pilot-source merge-base --is-ancestor "$PILOT_SHA" HEAD
git -C .runner-pilot-source checkout --detach "$PILOT_SHA"
- name: Install build dependencies
if: inputs.workload != 'probe'
shell: bash
run: |
sudo apt-get update
sudo apt-get install -y build-essential pkg-config libssl-dev cmake clang time
- name: Install the fixed protobuf compiler
if: inputs.workload != 'probe'
uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
with:
version: "35.1"
repo-token: ${{ github.token }}
- name: Install the fixed flatbuffer compiler
if: inputs.workload != 'probe'
uses: Nugine/setup-flatc@698800de72a96bfb22cf60431dc21a2ff9a7e07b # v1
with:
version: "25.12.19"
- name: Install the requested exact Rust toolchain
if: inputs.workload != 'probe'
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
with:
toolchain: ${{ inputs.toolchain }}
- name: Measure cold and warm builds without shared target caches
if: inputs.workload != 'probe'
shell: bash
env:
PILOT_WORKLOAD: ${{ inputs.workload }}
PILOT_SHA: ${{ inputs.source-sha }}
PILOT_TOOLCHAIN: ${{ inputs.toolchain }}
PILOT_SCRATCH: ${{ inputs.scratch-root }}
PILOT_JOBS: ${{ inputs.build-jobs }}
PILOT_SSD: ${{ inputs.local-ssd }}
run: |
extra=()
if [[ "$PILOT_SSD" == true ]]; then extra+=(--local-ssd-attestation /etc/rustfs-runner/storage.json); fi
python3 scripts/ci/runner_pilot.py --root "${PILOT_SCRATCH:-$RUNNER_TEMP}" \
--output "$RUNNER_TEMP/runner-pilot-evidence" \
--source .runner-pilot-source --source-sha "$PILOT_SHA" \
--toolchain "$PILOT_TOOLCHAIN" --benchmark "$PILOT_WORKLOAD" \
--jobs "$PILOT_JOBS" --timeout-seconds 3600 "${extra[@]}"
+5 -23
View File
@@ -66,7 +66,10 @@ inputs:
runs:
using: "composite"
steps:
# Use the pinned code generators rather than Ubuntu's older packages.
# protobuf-compiler is deliberately absent: the setup-protoc step below
# installs 35.1 into the tool cache and prepends it to PATH, so the apt
# build (older, and never version-matched) was shadowed on every run and
# simply never used.
- name: Install system dependencies (Ubuntu)
if: runner.os == 'Linux'
shell: bash
@@ -76,9 +79,7 @@ runs:
build-essential \
pkg-config \
libssl-dev \
ripgrep \
python3 \
gh
ripgrep
# musl-gcc is needed by the native musl release leg, and zip/unzip by the
# release packaging steps. No CI test lane touches any of them.
@@ -87,32 +88,13 @@ runs:
shell: bash
run: sudo apt-get install -y musl-tools zip unzip
- name: Install pinned code generators through GitHub API (Linux x64)
if: runner.os == 'Linux' && runner.arch == 'X64'
shell: bash
env:
GH_TOKEN: ${{ github.token }}
ASSET_DOWNLOADER: ${{ github.action_path }}/download_release_asset.py
run: |
python3 "$ASSET_DOWNLOADER" --repository protocolbuffers/protobuf --tag v35.1 --asset protoc-35.1-linux-x86_64.zip --output "$RUNNER_TEMP/protoc.zip"
python3 "$ASSET_DOWNLOADER" --repository google/flatbuffers --tag v25.12.19 --asset 'Linux.flatc.binary.g++-13.zip' --output "$RUNNER_TEMP/flatc.zip"
python3 -m zipfile -e "$RUNNER_TEMP/protoc.zip" "$RUNNER_TEMP/protoc"
python3 -m zipfile -e "$RUNNER_TEMP/flatc.zip" "$RUNNER_TEMP/flatc"
chmod +x "$RUNNER_TEMP/protoc/bin/protoc" "$RUNNER_TEMP/flatc/flatc"
"$RUNNER_TEMP/protoc/bin/protoc" --version
"$RUNNER_TEMP/flatc/flatc" --version
echo "$RUNNER_TEMP/protoc/bin" >> "$GITHUB_PATH"
echo "$RUNNER_TEMP/flatc" >> "$GITHUB_PATH"
- name: Install protoc
if: runner.os != 'Linux' || runner.arch != 'X64'
uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
with:
version: "35.1"
repo-token: ${{ github.token }}
- name: Install flatc
if: runner.os != 'Linux' || runner.arch != 'X64'
uses: Nugine/setup-flatc@698800de72a96bfb22cf60431dc21a2ff9a7e07b # v1
with:
version: "25.12.19"
@@ -1,84 +0,0 @@
#!/usr/bin/env python3
"""Download a named GitHub release asset through its API and verify its digest."""
import argparse
import hashlib
import json
from pathlib import Path
import re
import subprocess
import sys
import tempfile
from urllib.parse import quote
def download(repository, tag, name, output, expected_sha256=None):
if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]*/[A-Za-z0-9][A-Za-z0-9_.-]*", repository):
raise ValueError("invalid release repository")
if not tag or not name or "/" in name or "\\" in name:
raise ValueError("invalid release tag or asset name")
if expected_sha256 is not None and not re.fullmatch(r"[0-9a-f]{64}", expected_sha256):
raise ValueError("invalid pinned asset digest")
result = subprocess.run(
["gh", "api", "--hostname", "github.com", f"repos/{repository}/releases/tags/{quote(tag, safe='')}"],
check=True, capture_output=True, text=True, timeout=60,
)
release = json.loads(result.stdout)
if not isinstance(release, dict) or release.get("tag_name") != tag:
raise ValueError("release metadata does not match the requested tag")
assets = release.get("assets")
if not isinstance(assets, list) or any(not isinstance(asset, dict) for asset in assets):
raise ValueError("invalid release asset inventory")
matches = [asset for asset in assets if asset.get("name") == name]
if len(matches) != 1:
raise ValueError("release must contain exactly one named asset")
asset = matches[0]
asset_id, size, digest = asset.get("id"), asset.get("size"), asset.get("digest")
if type(asset_id) is not int or asset_id <= 0 or type(size) is not int or not 0 < size <= 512 * 1024 * 1024:
raise ValueError("invalid release asset identity or size")
if not isinstance(digest, str) or not re.fullmatch(r"sha256:[0-9a-f]{64}", digest):
raise ValueError("release asset has no valid SHA-256 digest")
sha256 = digest.removeprefix("sha256:")
if expected_sha256 is not None and expected_sha256 != sha256:
raise ValueError("release digest differs from the pinned digest")
output = Path(output)
output.parent.mkdir(parents=True, exist_ok=True)
temporary = None
try:
with tempfile.NamedTemporaryFile(dir=output.parent, prefix=".release-asset-", delete=False) as stream:
temporary = Path(stream.name)
# gh authenticates to the API; its HTTP client strips credentials on
# cross-host redirects. No browser URL or token is supplied in argv.
subprocess.run(
["gh", "api", "--hostname", "github.com", "--header", "Accept: application/octet-stream", f"repos/{repository}/releases/assets/{asset_id}"],
check=True, stdout=stream, timeout=300,
)
with temporary.open("rb") as stream:
actual = hashlib.file_digest(stream, "sha256").hexdigest()
if temporary.stat().st_size != size or actual != sha256:
raise ValueError("downloaded release asset failed size or SHA-256 verification")
temporary.replace(output)
print(f"Verified release asset: {repository} {tag} {name} sha256={actual}")
finally:
if temporary is not None:
temporary.unlink(missing_ok=True)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--repository", required=True)
parser.add_argument("--tag", required=True)
parser.add_argument("--asset", required=True)
parser.add_argument("--output", required=True, type=Path)
parser.add_argument("--sha256")
args = parser.parse_args()
try:
download(args.repository, args.tag, args.asset, args.output, args.sha256)
except (ValueError, KeyError, TypeError, json.JSONDecodeError, subprocess.SubprocessError, OSError) as error:
print(f"Release asset download failed: {error}", file=sys.stderr)
return 1
return 0
if __name__ == "__main__":
raise SystemExit(main())
-16
View File
@@ -1,16 +0,0 @@
name: Verification runtime
description: Install the runtime for prebuilt test archives without a target cache
runs:
using: composite
steps:
- name: Install runtime tools
shell: bash
run: |
sudo apt-get update
sudo apt-get install -y libssl-dev ripgrep
- name: Install Rust toolchain
uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
- name: Install cargo-nextest
uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
with:
tool: cargo-nextest@0.9.138
-2
View File
@@ -32,8 +32,6 @@ updates:
- "overtrue"
- "majinghe"
ignore:
- dependency-name: "pulsar"
versions: [ "6.9.x" ]
- dependency-name: "object_store"
versions: [ "0.13.x" ]
- dependency-name: "libunftp"
File diff suppressed because it is too large Load Diff
+4 -11
View File
@@ -5,7 +5,7 @@
## Billing facts
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, including the `macos-latest-xlarge` release lane used here, and standard runners in private repositories.
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
@@ -13,7 +13,6 @@ Every other label in this repository is capacity RustFS pays for. ARC pods and l
|---|---|---|---|
| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job |
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
| `macos-latest-xlarge` | RustFS | GitHub-hosted larger Apple Silicon runner | macOS release builds, deferred until Linux acceptance for candidates |
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
@@ -33,7 +32,7 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection uses `ubuntu-latest` so a lab outage cannot also stop its independent monitor. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it wraps verified candidate binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection stays on the operator-managed `smoke-testing` runner as part of the automation environment. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
@@ -53,7 +52,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
These are the patterns that have cost real money here without validating anything more:
- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`.
- Per-push workflows on `main`. A merge triggers the applicable `ci.yml` matrix and cache warming; `build.yml` development builds are scheduled or manual; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- Per-push workflows on `main`. A merge triggers the full `ci.yml` matrix, `cache-warm.yml` and the `build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- `apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images.
- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further.
- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
@@ -67,7 +66,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
```
Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` use paid capacity. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
## Checklist for a new or moved job
@@ -77,9 +76,3 @@ Rows using `macos-latest-xlarge` or a label starting with `sm-`, `dind-`, `ubicl
- A self-hosted `runs-on` carries a comment naming its reason from the placement rule.
- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other.
- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies.
## Release stages
The [candidate release pipeline](../../docs/operations/release-pipeline.md) builds Linux first and defers paid desktop capacity until acceptance. Daily development builds also select Linux; use manual platform selection for desktop development artifacts. All six platforms remain required before publication. The pipeline records and promotes the exact tested bytes. Builder workflows have no public-release credentials. Keep periodic development builds, PR verification and field-test chains separate when their contracts differ; do not re-enable disabled workflows as part of a source-file migration.
Larger runners remain billed for public repositories; see [GitHub runner billing](https://docs.github.com/en/actions/concepts/runners/larger-runners#billing). Actual spending comes from organization billing, not runner labels or an estimated build duration.
+8 -149
View File
@@ -23,16 +23,9 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -48,16 +41,9 @@ on:
- 'deny.toml'
- '.github/actions/**'
- '.github/workflows/**'
- 'scripts/release/**'
- 'scripts/security/generate_release_supply_chain_assets.py'
- '.config/release-validation-plan.json'
- '.github/scheduled-validations.json'
- 'scripts/test_helm_release_workflow.py'
- 'scripts/test_release_packages.py'
- 'scripts/test_release_images.py'
- 'Dockerfile*'
- 'entrypoint.sh'
- 'bootstrap/**'
- 'scripts/release/create_or_update_release.sh'
- 'scripts/release/package_service_scripts.sh'
- 'scripts/release/package_versions.sh'
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
@@ -146,38 +132,12 @@ jobs:
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
actions: read
outputs:
release_changed: ${{ steps.scope.outputs.release_changed }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Scope native packaging checks to release changes
id: scope
env:
BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
run: |
python3 - <<'PY'
import os, re, subprocess
event, base = os.environ['GITHUB_EVENT_NAME'], os.environ.get('BASE_SHA', '')
changed = event == 'workflow_dispatch'
if event in ('push', 'pull_request'):
if not re.fullmatch(r'[0-9a-f]{40}', base) or base == '0' * 40:
changed = True
else:
paths = subprocess.check_output(['git', 'diff', '--name-only', base, os.environ['GITHUB_SHA']], text=True).splitlines()
pattern = r'^(scripts/release/|scripts/test_release_|Dockerfile|entrypoint.sh$|bootstrap/|\.dockerignore$|\.config/release-validation-plan.json$|\.github/workflows/(audit|build|docker|package|helm-package|release-[^.]+)\.yml$)'
changed = any(re.match(pattern, path) for path in paths)
with open(os.environ['GITHUB_OUTPUT'], 'a') as output:
output.write(f'release_changed={str(changed).lower()}\n')
PY
- name: Report unpinned GitHub Actions
run: ./scripts/security/check_workflow_pins.sh --enforce
@@ -190,35 +150,8 @@ jobs:
- name: Check checkouts clear their credentials
run: ./scripts/security/check_persist_credentials.sh
- name: Check release candidate and promotion contracts
run: python3 -m unittest discover -s scripts/release -p 'test_*.py'
- name: Prepare artifact transport rehearsal
run: |
mkdir -p "$RUNNER_TEMP/release-transport"
printf '{"purpose":"transport contract, not release evidence"}\n' > "$RUNNER_TEMP/release-transport/transport.json"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-transport-contract-${{ github.run_attempt }}
path: ${{ runner.temp }}/release-transport/transport.json
if-no-files-found: error
retention-days: 1
- name: Verify actual GitHub artifact metadata and downloaded bytes
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c
run = c.github(f"repos/{c.REPOSITORY}/actions/runs/{os.environ['GITHUB_RUN_ID']}")
destination = Path(os.environ['RUNNER_TEMP']) / 'release-transport-downloaded'
c.download_artifact(run, f"release-transport-contract-{os.environ['GITHUB_RUN_ATTEMPT']}", destination)
c.require(c.read_json(destination / 'transport.json') == {'purpose': 'transport contract, not release evidence'}, 'artifact round trip changed')
PY
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
- name: Check tier evidence workflow isolation
run: ./scripts/security/check_tier_artifact_workflow.sh
@@ -229,80 +162,6 @@ jobs:
- name: Check package service lifecycle contract
run: ./scripts/test_package_service_scripts.sh
release-package-contract:
name: Release package contract
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install real package managers
run: |
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Exercise accepted and substituted payloads through both recipes
run: python3 scripts/test_release_packages.py
release-image-contract:
name: Release image contract (${{ matrix.variant }})
needs: workflow-pin-report
if: needs.workflow-pin-report.outputs.release_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 35
strategy:
fail-fast: false
matrix:
variant: [musl, gnu]
env:
RELEASE_IMAGE_VARIANT: ${{ matrix.variant }}
NO_PROXY: 127.0.0.1,localhost,::1
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install OCI tools
run: |
echo "RELEASE_IMAGE_OUTPUT=$RUNNER_TEMP/image-contract" >> "$GITHUB_ENV"
echo 'no_proxy=127.0.0.1,localhost,::1' >> "$GITHUB_ENV"
sudo apt-get update
sudo apt-get install -y skopeo
go version
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Exercise real OCI packaging, startup, registry retry and conflicts
run: python3 scripts/test_release_images.py
- name: Exercise amd64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0"
- name: Exercise arm64 OCI archive scanning
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: ${{ runner.temp }}/image-contract-scan/image-${{ matrix.variant }}-arm64.tar
format: sarif
output: ${{ runner.temp }}/image-contract/scan-${{ matrix.variant }}-arm64.sarif
exit-code: "0"
- name: Verify scan receipts before recording the archive
run: |
python3 - <<'PY'
import os, sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import deliver, test_candidate
deliver.image_record(test_candidate.inputs(), os.environ['RELEASE_IMAGE_VARIANT'], Path(os.environ['RELEASE_IMAGE_OUTPUT']))
PY
dependency-review:
name: Dependency Review
runs-on: ubuntu-latest
+467 -192
View File
@@ -12,32 +12,51 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Nightly development artifacts and reusable immutable-candidate builds.
# Public release writes live exclusively in release-publish.yml. Candidate callers
# select Linux or desktop and pin release inputs before any matrix starts.
# Build and Release Workflow
#
# This workflow builds RustFS binaries and automatically triggers Docker image builds.
#
# Flow:
# 1. Build binaries for multiple platforms
# 2. Upload binaries to OSS storage
# 3. Trigger docker.yml to build and push images using the uploaded binaries
#
# Platform scope:
# - Development builds come from the nightly schedule and manual dispatch and
# cover the full platform matrix; they feed the dev download channel
# (artifacts/rustfs/dev on R2, the *-main-latest packages). Tags build the
# release matrix. Main pushes do not build: at the usual merge rate most
# per-merge builds were cancelled by the next merge before publishing
# anything, and the dev channel does not need per-merge granularity.
#
# Manual Parameters:
# - publish: Publish binaries/releases for manual runs (default: false)
# - build_docker: Advisory only; manual runs do not trigger Docker publishing
# - platforms: Comma-separated platform IDs or 'all' (default: all)
name: Build and Release
on:
workflow_call:
inputs:
candidate_manifest:
description: Frozen candidate inputs; candidates never publish from this workflow
required: true
type: string
platforms:
description: linux, desktop, all, or an explicit platform list
required: true
type: string
push:
tags: [ "*.*.*" ]
schedule:
- cron: "13 1 * * *" # Nightly 01:13 UTC
workflow_dispatch:
inputs:
publish:
description: "Publish the development download channel (never a release)"
description: "Publish to download channels and GitHub Releases (false keeps only workflow artifacts)"
required: false
default: false
type: boolean
build_docker:
# Advisory only. docker.yml triggers on workflow_run and its job-level
# condition requires the triggering event to be a tag push, so a manual
# dispatch of this workflow never produces images regardless of this
# value. Publishing remains controlled by the explicit publish input.
description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
required: false
default: true
type: boolean
platforms:
description: "Comma-separated targets or 'all' (e.g. linux-x86_64-musl,macos-aarch64)"
required: false
@@ -46,11 +65,10 @@ on:
permissions:
contents: read
actions: read
concurrency:
group: ${{ github.workflow }}-${{ github.event_name == 'push' && github.ref || github.run_id }}
cancel-in-progress: false
cancel-in-progress: ${{ github.event_name == 'push' }}
env:
CARGO_TERM_COLOR: always
@@ -66,39 +84,19 @@ jobs:
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_publish: ${{ inputs.candidate_manifest == '' && github.ref == 'refs/heads/main' && (github.event_name == 'schedule' || inputs.publish == true) }}
should_publish: ${{ github.event_name != 'workflow_dispatch' || inputs.publish == true }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
candidate_id: ${{ steps.candidate.outputs.candidate_id }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ inputs.candidate_manifest != '' && fromJson(inputs.candidate_manifest).source_sha || github.sha }}
- name: Validate frozen candidate
if: inputs.candidate_manifest != ''
id: candidate
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
python3 - <<'PYINPUT'
import json, os, sys
sys.path.insert(0, 'scripts/release')
import candidate
value = candidate.validate_inputs(json.loads(os.environ['CANDIDATE_MANIFEST']))
candidate.write_json('candidate-inputs.json', value)
candidate.emit_output('candidate_id', value['candidate_id'])
PYINPUT
python3 scripts/release/candidate.py validate-inputs --inputs candidate-inputs.json --check-checkout
- name: Determine build strategy
id: check
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
run: |
should_build=false
build_type="none"
@@ -109,14 +107,8 @@ jobs:
# Get short SHA for all builds
short_sha=$(git rev-parse --short HEAD)
# Candidates contain the final product version but have no release
# authority. Their only publication is immutable workflow artifacts.
if [[ -n "$CANDIDATE_MANIFEST" ]]; then
should_build=true
build_type="candidate"
version=$(python3 -c 'import json; print(json.load(open("candidate-inputs.json"))["version"])')
# Determine build type based on trigger
elif [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
# Tag push - preview, release, or prerelease
should_build=true
tag_name="${GITHUB_REF#refs/tags/}"
@@ -163,46 +155,6 @@ jobs:
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
prepare-build-inputs:
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
console: ${{ steps.inputs.outputs.console }}
toolchain: ${{ steps.inputs.outputs.toolchain }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ inputs.candidate_manifest != '' && fromJson(inputs.candidate_manifest).source_sha || github.sha }}
- name: Resolve one Console and toolchain identity
id: inputs
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
shell: bash
run: |
set -euo pipefail
if [[ -z "$CANDIDATE_MANIFEST" ]]; then
gh api repos/rustfs/console/releases/latest > console-release.json
rustup update stable --no-self-update
fi
python3 - <<'PYINPUT'
import json, os, subprocess, sys
sys.path.insert(0, 'scripts/release')
import candidate
raw = os.environ['CANDIDATE_MANIFEST']
if raw:
frozen = candidate.validate_inputs(json.loads(raw))
console, toolchain = frozen['console'], frozen['toolchain']
else:
console = candidate.console_asset(candidate.read_json('console-release.json'))
toolchain = subprocess.check_output(['rustc', '+stable', '--version'], text=True).split()[1]
candidate.emit_output('console', json.dumps(console, separators=(',', ':')))
candidate.emit_output('toolchain', toolchain)
PYINPUT
# Build RustFS binaries
prepare-platform-matrix:
name: Prepare Platform Matrix
@@ -219,9 +171,7 @@ jobs:
env:
# via env, not interpolation: a dispatch input is free-form text and
# would otherwise be pasted into the script for bash to evaluate.
# Daily development builds cover Linux. Desktop remains available on
# manual dispatch and is required after candidate field acceptance.
RAW_PLATFORMS: ${{ inputs.platforms || 'linux' }}
RAW_PLATFORMS: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}
run: |
set -euo pipefail
@@ -243,14 +193,10 @@ jobs:
{"target_id":"linux-aarch64-musl","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"macos-aarch64","os":"macos-latest-xlarge","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
]}'
if [[ "$selected" == "linux" || "$selected" == "desktop" ]]; then
selected=$(jq -r --arg group "$selected" '.include[] | select(if $group == "linux" then .platform == "linux" else .platform != "linux" end) | .target_id' <<<"$all" | paste -sd ',' -)
fi
if [[ "${selected}" == "all" ]]; then
matrix="$(jq -c . <<<"${all}")"
else
@@ -278,13 +224,12 @@ jobs:
build-rustfs:
# Acceptance workflows select artifacts by this stable platform name.
name: Build RustFS (${{ matrix.target_id }})
needs: [ build-check, prepare-platform-matrix, prepare-build-inputs ]
needs: [ build-check, prepare-platform-matrix ]
if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
runs-on: ${{ matrix.os }}
timeout-minutes: 180
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTUP_TOOLCHAIN: ${{ needs.prepare-build-inputs.outputs.toolchain }}
# Release binaries ship without dial9 telemetry and therefore do not need
# --cfg tokio_unstable. Telemetry builds opt in explicitly (see
# `make build-profiling`); crates/obs/build.rs fails the compile if the
@@ -299,33 +244,11 @@ jobs:
with:
persist-credentials: false
fetch-depth: 0
ref: ${{ inputs.candidate_manifest != '' && fromJson(inputs.candidate_manifest).source_sha || github.sha }}
- name: Ensure Python for artifact verification
if: runner.os == 'Linux'
shell: bash
run: |
if ! command -v python3 >/dev/null 2>&1; then
sudo apt-get update
sudo apt-get install -y python3
fi
- name: Reuse completed candidate platform
id: reuse
if: needs.build-check.outputs.build_type == 'candidate'
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target_id }}
shell: bash
run: |
python3 scripts/release/pipeline.py reuse-platform --target "$TARGET"
- name: Setup Rust environment
if: steps.reuse.outputs.reused != 'true'
uses: ./.github/actions/setup
with:
rust-version: ${{ needs.prepare-build-inputs.outputs.toolchain }}
rust-version: stable
target: ${{ matrix.target }}
# rust-cache keys include OS and architecture but not the runner image.
# The Ubicloud aarch64 legs therefore use their own suffix so they
@@ -345,35 +268,128 @@ jobs:
install-test-tools: 'false'
- name: Download static console assets
if: steps.reuse.outputs.reused != 'true'
shell: bash
env:
GH_TOKEN: ${{ github.token }}
CONSOLE_IDENTITY: ${{ needs.prepare-build-inputs.outputs.console }}
GITHUB_TOKEN: ${{ github.token }}
run: |
python3 - <<'PYCONSOLE'
import json, os, pathlib, subprocess, zipfile
value = json.loads(os.environ['CONSOLE_IDENTITY'])
subprocess.run(['python3', '.github/actions/setup/download_release_asset.py',
'--repository', 'rustfs/console', '--tag', value['tag'], '--asset', value['name'],
'--sha256', value['sha256'], '--output', 'console.zip'], check=True)
destination = pathlib.Path('rustfs/static')
destination.mkdir(parents=True, exist_ok=True)
with zipfile.ZipFile('console.zip') as archive:
for member in archive.infolist():
path = pathlib.PurePosixPath(member.filename)
if path.is_absolute() or '..' in path.parts or '\\' in member.filename:
raise SystemExit('unsafe Console archive member')
archive.extractall(destination)
if not (destination / 'index.html').is_file():
raise SystemExit('Console archive has no index.html')
PYCONSOLE
mkdir -p ./rustfs/static
verify_sha256() {
local expected="$1"
local file="$2"
local actual=""
local actual_line=""
if command -v sha256sum >/dev/null 2>&1; then
actual_line=$(sha256sum -- "$file") || return 1
elif command -v shasum >/dev/null 2>&1; then
actual_line=$(shasum -a 256 -- "$file") || return 1
else
echo "No SHA-256 verification tool found" >&2
return 1
fi
actual="${actual_line%%[[:space:]]*}"
if [ "$actual" = "$expected" ]; then
echo "SHA256 verified OK: $actual"
return 0
else
echo "SHA256 mismatch: expected=$expected actual=$actual" >&2
return 1
fi
}
download_console_assets() {
local curl_bin="curl"
local python_bin="python3"
local console_api="https://api.github.com/repos/rustfs/console/releases/latest"
local console_json="console-release.json"
local console_url
local console_sha256
local curl_auth_args=()
console_tag=""
if [[ "${{ matrix.platform }}" == "windows" ]]; then
curl_bin="curl.exe"
else
chmod +w ./rustfs/static/LICENSE || true
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
python_bin="python"
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
echo "No Python interpreter found for release metadata parsing" >&2
return 1
fi
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
curl_auth_args=(-H "Authorization: Bearer ${GITHUB_TOKEN}" -H "X-GitHub-Api-Version: 2022-11-28")
fi
"$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
-o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
import json
import re
import sys
with open(sys.argv[1], encoding="utf-8") as handle:
release = json.load(handle)
tag = release.get("tag_name", "") or "unknown"
for asset in release.get("assets", []):
name = asset.get("name", "")
digest = asset.get("digest", "")
url = asset.get("browser_download_url", "")
if name.startswith("rustfs-console-") and name.endswith(".zip") and digest.startswith("sha256:") and url:
sha256 = digest.split(":", 1)[1]
if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
break
else:
raise SystemExit("no console zip asset with sha256 digest found")
PY
) || return 1
if [[ -z "$console_url" || -z "$console_sha256" ]]; then
echo "Console release metadata is missing URL or SHA-256 digest" >&2
return 1
fi
echo "Console release: ${console_tag}"
echo "Downloading console asset: ${console_url}"
"$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
verify_sha256 "$console_sha256" console.zip || return 2
unzip -o console.zip -d ./rustfs/static || return 2
}
if download_console_assets; then
rm -f console.zip console-release.json
else
status=$?
rm -f console.zip console-release.json
if [[ "$status" -eq 2 ]]; then
echo "Console asset integrity verification failed" >&2
fi
echo "Failed to download verified console assets" >&2
exit 1
fi
if [[ ! -s ./rustfs/static/index.html ]]; then
echo "Console asset archive is missing static/index.html" >&2
exit 1
fi
asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
- name: Build RustFS
if: steps.reuse.outputs.reused != 'true'
shell: bash
env:
RUSTFS_BUILD_VERSION: ${{ needs.build-check.outputs.build_type == 'candidate' && needs.build-check.outputs.version || '' }}
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }}
run: |
@@ -424,7 +440,7 @@ jobs:
fi
- name: Verify macOS binary has no dynamic liblzma dependency
if: steps.reuse.outputs.reused != 'true' && (matrix.platform == 'macos')
if: matrix.platform == 'macos'
shell: bash
run: |
binary="target/${{ matrix.target }}/release/rustfs"
@@ -435,7 +451,7 @@ jobs:
"$binary" --version
- name: Generate the CPU symbol catalog
if: steps.reuse.outputs.reused != 'true' && (endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope')
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
shell: bash
run: |
cargo metadata --locked --format-version 1 --filter-platform '${{ matrix.target }}' \
@@ -452,7 +468,6 @@ jobs:
"$release_dir/rustfs.cpu-symbol-catalog.json" > "$release_dir/rustfs.cpu-symbol-catalog.sha256"
- name: Create release package
if: steps.reuse.outputs.reused != 'true'
id: package
shell: bash
run: |
@@ -653,7 +668,7 @@ jobs:
echo "📊 Version: ${VERSION}"
- name: Verify packaged Linux binary
if: steps.reuse.outputs.reused != 'true' && (matrix.platform == 'linux')
if: matrix.platform == 'linux'
shell: bash
run: |
set -euo pipefail
@@ -682,7 +697,7 @@ jobs:
"$binary_dir/rustfs" --version
- name: Verify packaged console
if: steps.reuse.outputs.reused != 'true' && (matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu')
if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
shell: bash
run: |
set -euo pipefail
@@ -735,49 +750,15 @@ jobs:
cat "$log_file"
exit 1
- name: Verify native candidate and retain receipts
if: needs.build-check.outputs.build_type == 'candidate' && steps.reuse.outputs.reused != 'true'
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target_id }}
ARCHIVE: ${{ steps.package.outputs.package_file }}
shell: bash
run: |
python3 -c 'import os; from pathlib import Path; Path("inputs.json").write_text(os.environ["CANDIDATE_MANIFEST"])'
python3 scripts/release/verify.py native --inputs inputs.json --archive "$ARCHIVE" --target "$TARGET" --output .
- name: Record candidate artifact identity
if: steps.reuse.outputs.reused != 'true' && (needs.build-check.outputs.build_type == 'candidate')
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET_ID: ${{ matrix.target_id }}
PACKAGE_FILE: ${{ steps.package.outputs.package_file }}
shell: bash
run: |
python3 - <<'PYRECORD'
import json, os, sys
sys.path.insert(0, 'scripts/release')
import candidate
value = candidate.validate_inputs(json.loads(os.environ['CANDIDATE_MANIFEST']))
record = candidate.archive_record(os.environ['PACKAGE_FILE'], os.environ['TARGET_ID'], value)
candidate.write_json('record-' + os.environ['TARGET_ID'] + '.json', record)
PYRECORD
- name: Upload to GitHub artifacts
if: steps.reuse.outputs.reused != 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ needs.build-check.outputs.build_type == 'candidate' && format('candidate-{0}-{1}', needs.build-check.outputs.candidate_id, matrix.target_id) || steps.package.outputs.package_name }}
path: |
rustfs-*.zip
record-${{ matrix.target_id }}.json
identity-${{ matrix.target_id }}*
startup-${{ matrix.target_id }}.log
if-no-files-found: error
retention-days: ${{ (startsWith(github.ref, 'refs/tags/') || needs.build-check.outputs.build_type == 'candidate') && 30 || 7 }}
name: ${{ steps.package.outputs.package_name }}
path: "rustfs-*.zip"
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}
- name: Upload to Cloudflare R2
if: steps.reuse.outputs.reused != 'true' && (needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development'))
if: needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
@@ -833,18 +814,13 @@ jobs:
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Determine upload path based on build type. Release and prerelease
# builds only stage their bytes here: the official
# artifacts/rustfs/release/ path is written solely by the
# certification-gated promote-release-artifacts job, from these same
# staged objects. Development builds keep the uncertified dev channel
# and never touch official or latest channels.
# Determine upload path based on build type
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/dev/"
echo "📤 Uploading development build to R2 dev directory"
else
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
echo "📤 Staging release build for certification-gated promotion"
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"
echo "📤 Uploading release build to R2 release directory"
fi
# Upload all rustfs zip files to R2 using S3-compatible API
@@ -859,7 +835,6 @@ jobs:
echo "✅ Upload completed successfully"
# Build summary
build-summary:
name: Build Summary
@@ -894,9 +869,309 @@ jobs:
echo "This run does not trigger Docker publishing."
fi
# Create GitHub Release for every valid release tag, including previews
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
outputs:
release_id: ${{ steps.create.outputs.release_id }}
release_url: ${{ steps.create.outputs.release_url }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0
- name: Create GitHub Release
id: create
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
VERSION="${{ needs.build-check.outputs.version }}"
IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
TARGET_COMMITISH=$(git rev-parse --verify "refs/tags/${TAG}^{commit}")
# Determine release type for title
if [[ "$BUILD_TYPE" == "preview" ]]; then
RELEASE_TYPE="preview"
elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
if [[ "$TAG" == *"alpha"* ]]; then
RELEASE_TYPE="alpha"
elif [[ "$TAG" == *"beta"* ]]; then
RELEASE_TYPE="beta"
elif [[ "$TAG" == *"rc"* ]]; then
RELEASE_TYPE="rc"
else
RELEASE_TYPE="prerelease"
fi
else
RELEASE_TYPE="release"
fi
# Create release title
if [[ "$IS_PRERELEASE" == "true" ]]; then
TITLE="RustFS $VERSION (${RELEASE_TYPE})"
else
TITLE="RustFS $VERSION"
fi
./scripts/release/create_or_update_release.sh \
"$TAG" \
"$TARGET_COMMITISH" \
"$TITLE" \
"$IS_PRERELEASE"
# Prepare and upload release assets
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
merge-multiple: true
- name: Prepare release assets
id: prepare
shell: bash
run: |
VERSION="${{ needs.build-check.outputs.version }}"
TAG="${{ needs.build-check.outputs.version }}"
mkdir -p ./release-assets
# Copy and verify artifacts (including latest files created during build)
ASSETS_COUNT=0
for file in ./artifacts/*.zip; do
if [[ -f "$file" ]]; then
cp "$file" ./release-assets/
ASSETS_COUNT=$((ASSETS_COUNT + 1))
fi
done
if [[ $ASSETS_COUNT -eq 0 ]]; then
echo "❌ No artifacts found!"
exit 1
fi
cd ./release-assets
# Generate checksums for all files (including latest versions)
if compgen -G "*.zip" >/dev/null; then
sha256sum -- *.zip > SHA256SUMS
sha512sum -- *.zip > SHA512SUMS
fi
cd ..
python3 scripts/security/generate_release_supply_chain_assets.py \
--asset-dir ./release-assets \
--version "$VERSION" \
--tag "$TAG" \
--build-type "${{ needs.build-check.outputs.build_type }}" \
--repository "${{ github.repository }}" \
--ref "${{ github.ref }}" \
--sha "${{ github.sha }}" \
--run-id "${{ github.run_id }}" \
--run-attempt "${{ github.run_attempt }}"
cd ./release-assets
echo "📦 Prepared assets:"
ls -la
echo "🔢 Total asset count: $ASSETS_COUNT"
- name: Upload to GitHub Release
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
cd ./release-assets
# Upload all files
for file in *; do
if [[ -f "$file" ]]; then
echo "📤 Uploading $file..."
gh release upload "$TAG" "$file" --clobber
fi
done
echo "✅ All assets uploaded successfully"
# Update latest.json for every release tag (stable and prerelease): the
# project currently ships prerelease tags only, so gating this to stable
# left the version pointer permanently stale. release_type records whether
# the pointed-to version is a prerelease.
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Update latest.json
env:
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
shell: bash
run: |
if [[ -z "$OSS_ACCESS_KEY_ID" ]]; then
echo "⚠️ OSS credentials not available, skipping latest.json update"
exit 0
fi
VERSION="${{ needs.build-check.outputs.version }}"
TAG="${{ needs.build-check.outputs.version }}"
if [[ "${{ needs.build-check.outputs.build_type }}" == "prerelease" ]]; then
RELEASE_TYPE="prerelease"
else
RELEASE_TYPE="stable"
fi
# Install ossutil
OSSUTIL_VERSION="2.1.1"
OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-amd64"
OSSUTIL_SHA256="60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2"
curl --fail -L -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
printf '%s %s\n' "$OSSUTIL_SHA256" "$OSSUTIL_ZIP" | sha256sum -c -
unzip "$OSSUTIL_ZIP"
OSSUTIL_BIN="${RUNNER_TEMP}/ossutil"
mv "${OSSUTIL_DIR}/ossutil" "$OSSUTIL_BIN"
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
chmod +x "$OSSUTIL_BIN"
# Create latest.json
cat > latest.json << EOF
{
"version": "${VERSION}",
"tag": "${TAG}",
"release_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
"release_type": "${RELEASE_TYPE}",
"download_url": "https://github.com/${{ github.repository }}/releases/tag/${TAG}"
}
EOF
# Upload to OSS
"$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force
echo "✅ Updated latest.json for ${RELEASE_TYPE} release $VERSION"
# Publish release (remove draft status)
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
steps:
- name: Publish release
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
TAG="${{ needs.build-check.outputs.version }}"
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
RELEASE_ID="${{ needs.create-release.outputs.release_id }}"
# Publish the release and correct its channel state on retries.
# Only a stable final release may become GitHub Latest.
if [[ "$BUILD_TYPE" == "release" ]]; then
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=false \
-f make_latest=true >/dev/null
else
gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-F draft=false \
-F prerelease=true \
-f make_latest=false >/dev/null
fi
echo "🎉 Released $TAG successfully!"
echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"
# Remove the internal preview releases once the deliverable release is live.
# Only the Releases are deleted; the -preview.N tags stay so the validated
# commit remains traceable.
cleanup-preview-releases:
name: Cleanup Preview Releases
needs: [ build-check, publish-release ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
steps:
- name: Delete preview releases for this target
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
set -euo pipefail
TAG="${{ needs.build-check.outputs.version }}"
RELEASES_JSON="${RUNNER_TEMP}/releases.json"
# Fetch before filtering: a failed listing must abort here instead of
# looking like "nothing to clean up".
gh api --paginate "repos/${GITHUB_REPOSITORY}/releases?per_page=100" > "$RELEASES_JSON"
# Match only <target>-preview.<digits>. String operations, not a
# regex over the tag, so dots in the version cannot widen the match.
DELETED=0
while IFS= read -r preview_tag; do
[[ -n "$preview_tag" ]] || continue
echo "🧹 Deleting preview release $preview_tag (tag kept)"
gh release delete "$preview_tag" --repo "${GITHUB_REPOSITORY}" --yes
DELETED=$((DELETED + 1))
done < <(
jq -r --arg tag "$TAG" '
.[]
| select(.tag_name | startswith($tag + "-preview."))
| select(.tag_name | ltrimstr($tag + "-preview.") | test("^[0-9]+$"))
| .tag_name
' "$RELEASES_JSON"
)
if [[ "$DELETED" -eq 0 ]]; then
echo "ℹ️ No preview releases to clean up for $TAG"
else
echo "✅ Removed $DELETED preview release(s) for $TAG"
fi
alert-on-failure:
name: Alert on scheduled failure
needs: [build-check, prepare-build-inputs, prepare-platform-matrix, build-rustfs, build-summary]
needs: [build-check, prepare-platform-matrix, build-rustfs, build-summary]
if: >-
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+2 -2
View File
@@ -96,7 +96,7 @@ env:
RUST_BACKTRACE: 1
jobs:
# Readers: workspace-lint-and-build, workspace-tests (workspace and ILM), build-rustfs-debug-binary,
# Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
# e2e-tests, e2e-full.
warm-ci-dev:
name: Warm ci-dev
@@ -146,7 +146,7 @@ jobs:
if-no-files-found: error
# --all-targets covers the test binaries nextest builds, including
# e2e_test, which the workspace run excludes. The second build adds
# e2e_test, which test-and-lint's own run excludes. The second build adds
# the e2e-test-hooks feature resolution that build-rustfs-debug-binary uses
# and that no lint lane enables.
- name: Build ci-dev superset
@@ -1,60 +0,0 @@
name: CI Resource Accounting
on:
workflow_dispatch:
inputs:
since:
description: "Inclusive run creation timestamp, for example 2026-10-01T00:00:00Z"
required: true
type: string
until:
description: "Inclusive end timestamp; keep the window below 1000 runs"
required: true
type: string
permissions:
contents: read
actions: read
pull-requests: read
concurrency:
group: ci-resource-accounting
cancel-in-progress: false
jobs:
collect:
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Collect all workflow attempts in the window
env:
GH_TOKEN: ${{ github.token }}
ACCOUNTING_SINCE: ${{ inputs.since }}
ACCOUNTING_UNTIL: ${{ inputs.until }}
ACCOUNTING_REPOSITORY: ${{ github.repository }}
run: |
python3 scripts/ci/actions_cost_report.py \
--repository "$ACCOUNTING_REPOSITORY" \
--since "$ACCOUNTING_SINCE" --until "$ACCOUNTING_UNTIL" \
--output "$RUNNER_TEMP/ci-accounting.json" \
--summary "$RUNNER_TEMP/ci-accounting.md"
- name: Show available accounting summary
if: always()
run: |
if [[ -f "$RUNNER_TEMP/ci-accounting.md" ]]; then
cat "$RUNNER_TEMP/ci-accounting.md" >> "$GITHUB_STEP_SUMMARY"
fi
- name: Retain recomputable accounting evidence
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: ci-accounting-${{ github.run_id }}-${{ github.run_attempt }}
path: |
${{ runner.temp }}/ci-accounting.json
${{ runner.temp }}/ci-accounting.md
if-no-files-found: error
retention-days: 30
-92
View File
@@ -1,92 +0,0 @@
name: CI Runner Pilot
on:
workflow_dispatch:
inputs:
pool:
description: "Existing runner pool; no infrastructure is created"
required: true
type: choice
options: [hosted, arc]
default: hosted
workload:
description: "Probe only, or perform cold/warm fixed-source builds"
required: true
type: choice
options: [probe, server, workspace-tests]
default: probe
source_sha:
description: "Exact 40-character merged main SHA; required for builds"
type: string
toolchain:
description: "Exact Rust version such as 1.92.0; required for builds"
type: string
scratch_root:
description: "Existing absolute scratch directory; empty uses runner.temp"
type: string
build_jobs:
description: "Fixed Cargo concurrency for both comparisons (1-16)"
type: string
default: "2"
local_ssd:
description: "Require operator /etc/rustfs-runner/storage.json attestation"
type: boolean
default: false
permissions:
contents: read
concurrency:
group: ci-runner-pilot
cancel-in-progress: false
jobs:
hosted:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main' && inputs.pool == 'hosted'
runs-on: ubuntu-latest
timeout-minutes: 150
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/runner-pilot
with:
workload: ${{ inputs.workload }}
source-sha: ${{ inputs.source_sha }}
toolchain: ${{ inputs.toolchain }}
scratch-root: ${{ inputs.scratch_root }}
build-jobs: ${{ inputs.build_jobs }}
local-ssd: ${{ inputs.local_ssd }}
- name: Retain hosted pilot evidence
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: runner-pilot-hosted-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/runner-pilot-evidence
if-no-files-found: error
retention-days: 30
arc:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main' && inputs.pool == 'arc'
# Disk and hardware: measure the existing workspace build pool's storage.
runs-on: sm-standard-4
timeout-minutes: 150
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/runner-pilot
with:
workload: ${{ inputs.workload }}
source-sha: ${{ inputs.source_sha }}
toolchain: ${{ inputs.toolchain }}
scratch-root: ${{ inputs.scratch_root }}
build-jobs: ${{ inputs.build_jobs }}
local-ssd: ${{ inputs.local_ssd }}
- name: Retain ARC pilot evidence
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: runner-pilot-arc-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/runner-pilot-evidence
if-no-files-found: error
retention-days: 30
+277 -437
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -95,7 +95,7 @@ jobs:
- name: Run CLA Bot
if: github.event_name != 'merge_group'
uses: overtrue/cla-bot@4bd1387c19691168c42fcf5edfa17b15819b2ab3 # v0.1.1
uses: overtrue/cla-bot@5e5929846a60cfb42b16d0d38e135ef67d94fc88 # v0.0.9
with:
github-token: ${{ github.token }}
registry-token: ${{ steps.registry-token.outputs.token }}
+571 -71
View File
@@ -1,99 +1,599 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate images
on:
workflow_call:
inputs:
candidate_manifest:
type: string
required: true
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Docker Images Workflow
#
# This workflow builds Docker images using pre-built binaries from the build workflow.
#
# Trigger Types:
# 1. workflow_run: Automatically triggered when "Build and Release" workflow completes
# 2. workflow_dispatch: Manual trigger for standalone Docker builds
#
# Key Features:
# - Only triggers when Linux builds (x86_64 + aarch64) are successful
# - Independent of macOS/Windows build status
# - Uses workflow_run event for precise control
# - Only builds Docker images for releases and prereleases (development builds are skipped)
name: Docker Images
# Permissions needed for workflow_run event and Docker registry access
permissions:
contents: read
actions: read
packages: write
on:
# Automatically triggered when build workflow completes
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
# Manual trigger with same parameters for consistency
workflow_dispatch:
inputs:
push_images:
description: "Push images to registries"
required: false
default: true
type: boolean
version:
description: "Version to build (latest, v1.0.0, v1.0.0-alpha1, or v1.0.1-preview.1)"
required: false
default: "latest"
type: string
force_rebuild:
description: "Force rebuild even if binary exists (useful for testing)"
required: false
default: false
type: boolean
env:
CONCLUSION: ${{ github.event.workflow_run.conclusion }}
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
HEAD_SHA: ${{ github.event.workflow_run.head_sha }}
TRIGGERING_EVENT: ${{ github.event.workflow_run.event }}
DOCKERHUB_USERNAME: rustfs
CARGO_TERM_COLOR: always
REGISTRY_DOCKERHUB: rustfs/rustfs
REGISTRY_GHCR: ghcr.io/${{ github.repository }}
REGISTRY_QUAY: quay.io/rustfs/rustfs
DOCKER_PLATFORMS: linux/amd64,linux/arm64
jobs:
build-docker:
# Check if we should build Docker images.
# Short-circuit at job level so per-merge development builds (workflow_run
# from a push to main) skip the whole run without spawning a checkout job:
# images are only published for tag builds — build.yml push triggers cover
# only main and tags, so a non-main head_branch is a tag name — and for
# manual dispatch.
build-check:
name: Docker Build Check
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
timeout-minutes: 90
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_push: ${{ steps.check.outputs.should_push }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
create_latest: ${{ steps.check.outputs.create_latest }}
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
- name: Check build conditions
id: check
env:
# dispatch inputs via env, not `${{ }}` interpolation: they are
# free-form strings and would otherwise be evaluated by bash.
INPUT_VERSION: ${{ github.event.inputs.version }}
INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
run: |
should_build=false
should_push=false
build_type="none"
version=""
short_sha=""
is_prerelease=false
create_latest=false
source_ref="$GITHUB_SHA"
# Pre-GA policy: until the first stable (vX.Y.Z) tag exists, every
# prerelease (alpha/beta/rc) also moves `latest`, so users pulling
# `latest` get the newest test build. Once a stable tag is published
# this returns false and `latest` follows stable releases only.
prerelease_moves_latest() {
local stable_tags
stable_tags=$(git ls-remote --tags --refs origin 2>/dev/null \
| awk '{print $2}' \
| grep -E '^refs/tags/v?[0-9]+\.[0-9]+\.[0-9]+$' || true)
if [[ -z "$stable_tags" ]]; then
return 0
fi
echo "ℹ️ Stable release tag(s) already exist; prereleases no longer update latest"
return 1
}
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Triggered by build workflow completion
echo "🔗 Triggered by build workflow completion"
# Check if the triggering workflow was successful
# If the workflow succeeded, it means ALL builds (including Linux x86_64 and aarch64) succeeded
if [[ "$CONCLUSION" == "success" ]]; then
echo "✅ Build workflow succeeded, all builds including Linux are successful"
should_build=true
should_push=true
else
echo "❌ Build workflow failed (conclusion: $CONCLUSION), skipping Docker build"
should_build=false
fi
# Extract version info from commit message or use commit SHA
# Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
short_sha=$(git rev-parse --short "$HEAD_SHA")
source_ref="$HEAD_SHA"
# Determine build type based on triggering workflow event and ref
triggering_event="$TRIGGERING_EVENT"
head_branch="$HEAD_BRANCH"
echo "🔍 Analyzing triggering workflow:"
echo " 📋 Event: $triggering_event"
echo " 🌿 Head branch: $head_branch"
echo " 📎 Head SHA: $HEAD_SHA"
# Check if this was triggered by a tag push
if [[ "$triggering_event" == "push" ]]; then
# For tag pushes, head_branch will be like "refs/tags/v1.0.0" or just "v1.0.0"
if [[ "$head_branch" == refs/tags/* ]]; then
# Extract tag name from refs/tags/TAG_NAME
tag_name="${head_branch#refs/tags/}"
version="$tag_name"
elif [[ "$head_branch" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+ ]]; then
# Direct tag name like "v1.0.0" or "1.0.0-alpha.1"
version="$head_branch"
elif [[ "$head_branch" == "main" ]]; then
# Regular branch push to main
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (main branch push)"
else
# Other branch push
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (branch: $head_branch)"
fi
# If we extracted a version (tag), determine release type
if [[ -n "$version" ]] && [[ "$version" != "dev-${short_sha}" ]]; then
# Remove 'v' prefix if present for consistent version format
if [[ "$version" == v* ]]; then
version="${version#v}"
fi
if [[ "$version" =~ -preview\.[0-9]+$ ]]; then
build_type="preview"
is_prerelease=true
echo "🔍 Building Docker image for preview: $version (version tags only)"
elif [[ "$version" == *"-preview"* ]]; then
echo "❌ Invalid preview tag: $version (expected suffix: -preview.<number>)" >&2
exit 1
elif [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
else
echo "🧪 Building Docker image for prerelease: $version"
fi
else
build_type="release"
create_latest=true
echo "🚀 Building Docker image for release: $version"
fi
fi
else
# Non-push events
build_type="development"
version="dev-${short_sha}"
should_build=false
echo "⏭️ Skipping Docker build for development version (event: $triggering_event)"
fi
echo "🔄 Build triggered by workflow_run:"
echo " 📋 Conclusion: $CONCLUSION"
echo " 🌿 Branch: $HEAD_BRANCH"
echo " 📎 SHA: $HEAD_SHA"
echo " 🎯 Event: $TRIGGERING_EVENT"
elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Manual trigger
input_version="$INPUT_VERSION"
version="${input_version}"
should_push="$INPUT_PUSH_IMAGES"
should_build=true
# Get short SHA
short_sha=$(git rev-parse --short HEAD)
echo "🎯 Manual Docker build triggered:"
echo " 📋 Requested version: $input_version"
echo " 🔧 Force rebuild: $INPUT_FORCE_REBUILD"
echo " 🚀 Push images: $should_push"
case "$input_version" in
"latest")
build_type="release"
create_latest=true
echo "🚀 Building with latest stable release version"
;;
*-preview*)
if [[ ! "$input_version" =~ ^v?[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?-preview\.[0-9]+$ ]]; then
echo "❌ Invalid preview version: $input_version" >&2
exit 1
fi
build_type="preview"
is_prerelease=true
version="${input_version#v}"
echo "🔍 Building with preview version: $version (version tags only)"
;;
# Prerelease versions (must match first, more specific)
v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
build_type="prerelease"
is_prerelease=true
# Pre-GA policy: prereleases update latest until the first stable tag exists.
if prerelease_moves_latest; then
create_latest=true
echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
else
echo "🧪 Building with prerelease version: $input_version"
fi
;;
# Release versions (match after prereleases, more general)
v[0-9]*|[0-9]*.*.*)
build_type="release"
create_latest=true
echo "📦 Building with specific release version: $input_version"
;;
*)
# Invalid version for Docker build
should_build=false
echo "❌ Invalid version for Docker build: $input_version"
echo "⚠️ Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
;;
esac
if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
tag_ref="refs/tags/$input_version"
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
if [[ "$input_version" == v* ]]; then
tag_ref="refs/tags/${input_version#v}"
else
tag_ref="refs/tags/v$input_version"
fi
fi
if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
echo "❌ Release tag not found for Docker build: $input_version"
exit 1
fi
source_ref="$tag_ref"
fi
fi
{
echo "should_build=$should_build"
echo "should_push=$should_push"
echo "build_type=$build_type"
echo "version=$version"
echo "short_sha=$short_sha"
echo "is_prerelease=$is_prerelease"
echo "create_latest=$create_latest"
echo "source_ref=$source_ref"
} >> "$GITHUB_OUTPUT"
echo "🐳 Docker Build Summary:"
echo " - Should build: $should_build"
echo " - Should push: $should_push"
echo " - Build type: $build_type"
echo " - Version: $version"
echo " - Short SHA: $short_sha"
echo " - Is prerelease: $is_prerelease"
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
include:
- variant: musl
file: Dockerfile
- variant: gnu
suffix: ""
- variant: glibc
file: Dockerfile.glibc
env:
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
VARIANT: ${{ matrix.variant }}
suffix: "-glibc"
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Reuse completed image
id: reuse
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py reuse --kind image --target "$VARIANT"
- name: Install OCI tools
if: steps.reuse.outputs.reused != 'true'
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Login to Docker Hub
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Login to GitHub Container Registry
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Login to Quay.io
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Set up QEMU
uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
- name: Extract metadata and generate tags
id: meta
run: |
sudo apt-get update
sudo apt-get install -y skopeo
- uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3
if: steps.reuse.outputs.reused != 'true'
- uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
if: steps.reuse.outputs.reused != 'true'
- name: Supply accepted binaries and pinned CLI
if: steps.reuse.outputs.reused != 'true'
env:
GH_TOKEN: ${{ github.token }}
run: python3 scripts/release/deliver.py image-prepare --target "$VARIANT"
- name: Build OCI archive without registry writes
if: steps.reuse.outputs.reused != 'true'
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
VARIANT_SUFFIX="${{ matrix.suffix }}"
# Convert version format for Dockerfile compatibility. The former
# DOCKER_CHANNEL was "release" down every branch and was passed as a
# build-arg no Dockerfile declares, so it is gone.
case "$VERSION" in
"latest")
DOCKER_RELEASE="latest"
;;
v*)
# For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
DOCKER_RELEASE="${VERSION#v}"
;;
*)
# For other versions, pass as-is
DOCKER_RELEASE="${VERSION}"
;;
esac
echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
echo "🐳 Docker build parameters:"
echo " - Original version: $VERSION"
echo " - Docker RELEASE: $DOCKER_RELEASE"
# Generate tags based on build type
# Only support release and prerelease builds (no development builds)
TAG_BASE="${VERSION}${VARIANT_SUFFIX}"
TAGS="${{ env.REGISTRY_DOCKERHUB }}:$TAG_BASE,${{ env.REGISTRY_GHCR }}:$TAG_BASE,${{ env.REGISTRY_QUAY }}:$TAG_BASE"
# Add latest when requested (stable releases, and prereleases before GA)
if [[ "$CREATE_LATEST" == "true" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
fi
# Always add the channel tag for prereleases, independent of latest
if [[ "$BUILD_TYPE" == "prerelease" ]]; then
# Prerelease channel tags (alpha, beta, rc)
if [[ "$VERSION" == *"alpha"* ]]; then
CHANNEL="alpha"
elif [[ "$VERSION" == *"beta"* ]]; then
CHANNEL="beta"
elif [[ "$VERSION" == *"rc"* ]]; then
CHANNEL="rc"
fi
if [[ -n "$CHANNEL" ]]; then
TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:${CHANNEL}${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:${CHANNEL}${VARIANT_SUFFIX}"
fi
fi
# Output tags
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
SOURCE_REVISION="$(git rev-parse HEAD)"
LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
echo "labels=$LABELS" >> "$GITHUB_OUTPUT"
echo "🐳 Generated Docker tags:"
echo "$TAGS" | tr ',' '\n' | sed 's/^/ - /'
echo "📋 Build type: $BUILD_TYPE"
echo "🔖 Version: $VERSION"
- name: Build and push Docker image
uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
with:
context: .
file: ${{ matrix.file }}
platforms: linux/amd64,linux/arm64
push: false
provenance: false
outputs: type=oci,dest=delivery/image-${{ matrix.variant }}.tar
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
# minute or two those take, while creating a correctness problem: with
# RELEASE=latest the binary URL is resolved by curl *inside* a RUN
# layer, and the layer key does not include what that resolved to. A
# rebuild at the same RELEASE value (dispatch with version=latest, or
# a re-run of the same version) would hit the old layer and ship the
# previous release's binary. mode=max also consumed the same 10GB
# Actions cache quota the Rust lanes are fighting over.
#
# Only RELEASE is passed: it is the sole build-arg the Dockerfiles
# declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
# and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
# was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
# are declared by the Dockerfiles but deliberately left unset —
# supplying them would change the published image labels.
build-args: |
BINARY_SOURCE=supplied
RELEASE=${{ fromJson(inputs.candidate_manifest).version }}
VCS_REF=${{ fromJson(inputs.candidate_manifest).source_sha }}
- name: Verify wrapped binaries and startup on both architectures
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-smoke --target "$VARIANT"
- name: Scan amd64 image before sealing
if: steps.reuse.outputs.reused != 'true'
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/amd64
RELEASE=${{ steps.meta.outputs.docker_release }}
provenance: true
sbom: true
# Add retry mechanism by splitting the build process
no-cache: false
pull: true
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
# Report-only container image vulnerability scan
scan-docker-image:
name: Scan Docker Images
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
include:
- variant: musl
suffix: ""
- variant: glibc
suffix: "-glibc"
steps:
- name: Login to GitHub Container Registry
# docker/login-action v3
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
with:
input: delivery-scan/image-${{ matrix.variant }}-amd64.tar
format: sarif
output: delivery/scan-${{ matrix.variant }}-amd64.sarif
exit-code: "0" # Keep the repository's report-only vulnerability policy.
- name: Scan arm64 image before sealing
if: steps.reuse.outputs.reused != 'true'
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- name: Scan image with Trivy
# aquasecurity/trivy-action v0.36.0
uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
env:
TRIVY_PLATFORM: linux/arm64
with:
input: delivery-scan/image-${{ matrix.variant }}-arm64.tar
image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}
format: sarif
output: delivery/scan-${{ matrix.variant }}-arm64.sarif
output: trivy-${{ matrix.variant }}.sarif
ignore-unfixed: true
vuln-type: os,library
severity: CRITICAL,HIGH
exit-code: "0"
- name: Record verified OCI identity
if: steps.reuse.outputs.reused != 'true'
run: python3 scripts/release/deliver.py image-record --target "$VARIANT"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
# Surface findings in the Security tab; an artifact alone is write-only
# reporting nobody reviews.
- name: Upload scan results to code scanning
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
with:
name: image-${{ matrix.variant }}
path: delivery/*
sarif_file: trivy-${{ matrix.variant }}.sarif
category: container-image-${{ matrix.variant }}
- name: Upload container scan report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: container-image-scan-${{ matrix.variant }}
path: trivy-${{ matrix.variant }}.sarif
if-no-files-found: error
retention-days: 30
compression-level: 0
# Docker build summary
docker-summary:
name: Docker Build Summary
needs: [ build-check, build-docker ]
if: always() && needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Docker build completion summary
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
echo "🐳 Docker build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo "🚀 Strategy: Images using pre-built binaries (release channel only)"
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview Docker image has been built with ${VERSION} tags"
echo "⏭️ Preview images do not update latest or prerelease channels"
;;
"release")
echo "🚀 Release Docker image has been built with ${VERSION} tags"
echo "✅ This image is ready for production use"
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for stable release"
fi
;;
"prerelease")
echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
echo "⚠️ This is a prerelease image - use with caution"
# Prereleases move latest until the first stable tag exists (pre-GA policy).
if [[ "$CREATE_LATEST" == "true" ]]; then
echo "🏷️ Latest tag has been created for prerelease: $VERSION"
else
echo "🚫 Latest tag NOT created for prerelease"
fi
;;
*)
echo "❌ Unexpected build type: $BUILD_TYPE"
;;
esac
+28 -2
View File
@@ -17,8 +17,8 @@
# Each selected test starts a real localhost cluster via
# `RustFSTestClusterEnvironment` (4 processes; 4 drives per node unless the
# case is a two-site 4-node 1-drive pair or a 4-node upgrade). Membership is
# `[profile.e2e-distributed]` in `.config/nextest.toml`. The nightly schedule
# and manual dispatches execute the same fail-closed suite.
# `[profile.e2e-distributed]` in `.config/nextest.toml`. Storage-sensitive PRs,
# nightly runs, and manual dispatches all execute the same fail-closed suite.
# Upgrade cases download the same pinned previous release as e2e-upgrade.yml.
#
# Isolated pool filesystems: expand/decommission/rebalance cases require
@@ -33,6 +33,32 @@
name: e2e-distributed
on:
pull_request:
paths:
- "Cargo.lock"
- "Cargo.toml"
- ".config/nextest.toml"
- ".config/e2e-distributed-selection.txt"
- ".github/workflows/e2e-distributed.yml"
- "crates/audit/**"
- "crates/common/**"
- "crates/config/**"
- "crates/e2e_test/**"
- "crates/ecstore/**"
- "crates/filemeta/**"
- "crates/heal/**"
- "crates/iam/**"
- "crates/lock/**"
- "crates/madmin/**"
- "crates/notify/**"
- "crates/replication/**"
- "crates/s3-client/**"
- "crates/s3-ops/**"
- "crates/s3-types/**"
- "crates/scanner/**"
- "crates/storage-api/**"
- "crates/utils/**"
- "rustfs/**"
workflow_dispatch:
inputs:
filter:
+18 -7
View File
@@ -12,19 +12,30 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Single-node upgrade and rollback lane.
#
# The weekly Monday schedule, version tags, and workflow_dispatch run this
# suite. Distributed upgrade cases stay on the nightly e2e-distributed lane.
name: Upgrade Compatibility
on:
pull_request:
paths:
- ".github/workflows/e2e-upgrade.yml"
- "crates/e2e_test/src/common.rs"
- "crates/e2e_test/src/distributed/**"
- "crates/e2e_test/src/fake_s3_target/**"
- "crates/e2e_test/src/lib.rs"
- "crates/e2e_test/src/replication_extension_test.rs"
- "crates/e2e_test/src/rpc_fault_proxy.rs"
- "crates/e2e_test/src/upgrade_compatibility_test.rs"
- "crates/ecstore/**"
- "crates/filemeta/**"
- "crates/heal/**"
- "crates/kms/**"
- "crates/storage-api/**"
- "rustfs/**"
- "Cargo.lock"
push:
tags:
- "[0-9]*.[0-9]*.[0-9]*"
schedule:
# 03:17 UTC every Monday.
- cron: "17 3 * * 1"
workflow_dispatch:
@@ -33,7 +44,7 @@ permissions:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: false
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
env:
CARGO_TERM_COLOR: always
+33 -2
View File
@@ -12,13 +12,44 @@ concurrency:
jobs:
collect:
if: github.repository == 'rustfs/rustfs'
# Health remains available when either laboratory fleet is offline.
runs-on: ubuntu-latest
runs-on: smoke-testing
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Collect and publish verified chain health
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
+183 -50
View File
@@ -1,26 +1,136 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate Helm chart
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Publish helm chart to artifacthub
on:
workflow_call:
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
inputs:
candidate_manifest:
type: string
version:
description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
required: true
default: "1.0.0-beta.1"
type: string
permissions:
contents: read
actions: read
jobs:
package:
build-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 15
timeout-minutes: 30
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
contains(github.event.workflow_run.head_branch, '.') &&
!contains(github.event.workflow_run.head_branch, '-preview')
)
outputs:
raw_tag: ${{ steps.version.outputs.raw_tag }}
app_version: ${{ steps.version.outputs.app_version }}
chart_version: ${{ steps.version.outputs.chart_version }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- name: Checkout release source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
persist-credentials: false
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned yq
# Both inputs reach the shell through env rather than `${{ }}`
# interpolation. A git ref name may contain `$(...)` — anything without a
# space is a legal tag — and interpolation pastes it into the script
# verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
# data.
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -eux
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*)
RAW_TAG="${RAW#refs/tags/}"
;;
*)
RAW_TAG="$RAW"
;;
esac
./scripts/helm_chart_version.sh "$RAW_TAG"
# Docker and Helm share the binary-build trigger. Do not publish a chart
# until the image selected by its default appVersion is pullable.
- name: Wait for release image
timeout-minutes: 20
env:
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -euo pipefail
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
[[ "$APP_VERSION" == *-preview* ]]; then
echo "Invalid deliverable image version: $APP_VERSION" >&2
exit 1
fi
for attempt in {1..60}; do
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
| jq -er '.token // empty') &&
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
--header "Authorization: Bearer $token" \
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
exit 0
fi
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
if (( attempt < 60 )); then
sleep 15
fi
done
echo "Release image is unavailable; refusing to publish Helm chart" >&2
exit 1
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -eux
sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install yq
run: |
set -euo pipefail
yq_version="4.54.1"
@@ -35,43 +145,66 @@ jobs:
echo "$tools_dir" >> "$GITHUB_PATH"
"$tools_dir/yq" --version
- name: Test and package the exact application version
id: chart
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
- name: Test Helm Chart Templates
run: ./scripts/test_helm_templates.sh
- name: Package Helm Chart
run: |
python3 - <<'PYCHART'
import sys, subprocess, shutil
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline
inputs = pipeline.manifest_env()
current = pipeline.current_run('release-complete.yml')
if any(a['name'] == 'helm-package' for a in c.artifact_list(current['id'])):
c.download_artifact(current, 'helm-package', 'chart')
record = c.read_json('chart/chart.json')
c.require(record['inputs_sha256'] == c.digest(inputs), 'chart belongs to another candidate')
c.emit_output('reused', 'true')
else:
chart = Path('helm/rustfs/Chart.yaml')
import re
value = chart.read_text()
for key in ('version', 'appVersion'):
value, count = re.subn(rf'(?m)^{key}:.*$', f'{key}: "{inputs["version"]}"', value)
c.require(count == 1, 'ambiguous chart metadata')
chart.write_text(value)
subprocess.run(['bash', 'scripts/test_helm_templates.sh'], check=True)
shutil.copyfile('helm/README.md', 'helm/rustfs/README.md')
subprocess.run(['helm', 'package', 'helm/rustfs', '--destination', 'chart'], check=True)
path = Path('chart') / f"rustfs-{inputs['version']}.tgz"
c.write_json('chart/chart.json', {'version': inputs['version'], 'inputs_sha256': c.digest(inputs),
'name': path.name, 'sha256': c.file_digest(path), 'size': path.stat().st_size})
PYCHART
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.chart.outputs.reused != 'true'
set -eux
cp helm/README.md helm/rustfs/
helm package ./helm/rustfs \
--destination helm/rustfs/ \
--version "${{ steps.version.outputs.chart_version }}"
- name: Upload helm package as artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: helm-package
path: chart/*
if-no-files-found: error
retention-days: 30
path: |
helm/rustfs/*.tgz
helm/rustfs/README.md
retention-days: 1
publish-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ build-helm-package ]
if: needs.build-helm-package.result == 'success'
steps:
- name: Checkout helm package repo
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
repository: rustfs/helm
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: helm-package
path: ./
- name: Set up helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Generate index
run: helm repo index . --url https://charts.rustfs.com
# app_version is derived from the triggering tag name, and this job holds
# the cross-repository push token with rustfs/helm already checked out —
# the worst place in the repo to paste an attacker-influenced string into
# a shell line. Passed through env so bash treats it as data.
- name: Push helm package and index file
env:
GIT_USERNAME: ${{ secrets.USERNAME }}
GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
run: |
set -eux
git config --global user.name "${GIT_USERNAME}"
git config --global user.email "${GIT_EMAIL}"
git add .
git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
git push origin main
+7 -279
View File
@@ -87,206 +87,18 @@ jobs:
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Download static console assets
shell: bash
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
# Packaging tools are off on this lane, so setup does not install unzip.
if ! command -v unzip >/dev/null 2>&1; then
sudo apt-get update -qq
sudo apt-get install -y -qq unzip
fi
if ! command -v unzip >/dev/null 2>&1; then
echo "unzip is required to extract console assets" >&2
exit 1
fi
mkdir -p ./rustfs/static
verify_sha256() {
local expected="$1"
local file="$2"
local actual=""
local actual_line=""
if command -v sha256sum >/dev/null 2>&1; then
actual_line=$(sha256sum -- "$file") || return 1
elif command -v shasum >/dev/null 2>&1; then
actual_line=$(shasum -a 256 -- "$file") || return 1
else
echo "No SHA-256 verification tool found" >&2
return 1
fi
actual="${actual_line%%[[:space:]]*}"
if [ "$actual" = "$expected" ]; then
echo "SHA256 verified OK: $actual"
return 0
else
echo "SHA256 mismatch: expected=$expected actual=$actual" >&2
return 1
fi
}
download_console_assets() {
local curl_bin="curl"
local python_bin="python3"
local console_api="https://api.github.com/repos/rustfs/console/releases/latest"
local console_json="console-release.json"
local console_url
local console_sha256
local curl_auth_args=()
console_tag=""
chmod +w ./rustfs/static/LICENSE || true
if ! command -v "$python_bin" >/dev/null 2>&1; then
python_bin="python"
fi
if ! command -v "$python_bin" >/dev/null 2>&1; then
echo "No Python interpreter found for release metadata parsing" >&2
return 1
fi
if [[ -n "${GITHUB_TOKEN:-}" ]]; then
curl_auth_args=(-H "Authorization: Bearer ${GITHUB_TOKEN}" -H "X-GitHub-Api-Version: 2022-11-28")
fi
"$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
-o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
import json
import re
import sys
with open(sys.argv[1], encoding="utf-8") as handle:
release = json.load(handle)
tag = release.get("tag_name", "") or "unknown"
for asset in release.get("assets", []):
name = asset.get("name", "")
digest = asset.get("digest", "")
url = asset.get("browser_download_url", "")
if name.startswith("rustfs-console-") and name.endswith(".zip") and digest.startswith("sha256:") and url:
sha256 = digest.split(":", 1)[1]
if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
break
else:
raise SystemExit("no console zip asset with sha256 digest found")
PY
) || return 1
if [[ -z "$console_url" || -z "$console_sha256" ]]; then
echo "Console release metadata is missing URL or SHA-256 digest" >&2
return 1
fi
echo "Console release: ${console_tag}"
echo "Downloading console asset: ${console_url}"
"$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
verify_sha256 "$console_sha256" console.zip || return 2
unzip -o console.zip -d ./rustfs/static || return 2
}
if download_console_assets; then
rm -f console.zip console-release.json
else
status=$?
rm -f console.zip console-release.json
if [[ "$status" -eq 2 ]]; then
echo "Console asset integrity verification failed" >&2
fi
echo "Failed to download verified console assets" >&2
exit 1
fi
if [[ ! -s ./rustfs/static/index.html ]]; then
echo "Console asset archive is missing static/index.html" >&2
exit 1
fi
asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
- name: Build RustFS
# Pin the glibc baseline (`<target>.<glibc>` for cargo-zigbuild) so
# nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
shell: bash
run: |
# Force rebuild by touching build.rs so a restored cache cannot
# reuse a rustfs artifact compiled before these console assets existed.
touch rustfs/build.rs
cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bin rustfs
run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins
- name: Verify glibc symbol baseline
run: |
python3 scripts/release/check_glibc.py 2.31 \
target/x86_64-unknown-linux-gnu/release/rustfs
target/x86_64-unknown-linux-gnu/release/rustfs \
target/x86_64-unknown-linux-gnu/release/rustfs-cli
target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu'
- name: Verify console
shell: bash
run: |
set -euo pipefail
find_free_port() {
python3 - <<'PY'
import socket
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
print(sock.getsockname()[1])
PY
}
binary="target/x86_64-unknown-linux-gnu/release/rustfs"
data_dir="$(mktemp -d)"
log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
server_pid=""
trap '
if [[ -n "${server_pid:-}" ]]; then
kill "$server_pid" 2>/dev/null || true
wait "$server_pid" 2>/dev/null || true
fi
rm -rf "$data_dir"
' EXIT
if [[ ! -x "$binary" ]]; then
echo "rustfs binary not found: ${binary}" >&2
exit 1
fi
api_port="$(find_free_port)"
console_port="$(find_free_port)"
console_url="http://127.0.0.1:${console_port}/rustfs/console/"
"$binary" server \
--address "127.0.0.1:${api_port}" \
--console-enable \
--console-address "127.0.0.1:${console_port}" \
--access-key console-smoke \
--secret-key console-smoke-secret \
"$data_dir" >"$log_file" 2>&1 &
server_pid=$!
response=""
for _ in {1..40}; do
response="$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)"
if [[ "$response" == 200\ text/html* ]]; then
exit 0
fi
sleep 0.25
done
echo "Console endpoint did not return 200 text/html: $response" >&2
cat "$log_file"
exit 1
- name: Build DEB package
id: deb
shell: bash
@@ -534,12 +346,12 @@ jobs:
path: ${{ steps.publish.outputs.candidate_file }}
if-no-files-found: error
# Publish the deb/rpm pair to the auto-testing repo's `assets` branch so
# engineers can download and install the nightly directly. The branch is
# a single-commit orphan rewritten on every build, which keeps the repo
# small while the latest files stay reachable at stable raw URLs.
# Publish the deb/rpm pair as assets of the rolling `nightly` release on
# rustfs/auto-testing (see scripts/release/publish_nightly_assets.sh).
# The dispatch below runs only after this upload and the immutable R2
# candidate both succeed. It does not replace rustfs-functional-chain.yml,
# which still starts from this workflow's workflow_run completion.
# kms-vault-lane and kms-vault-ha-failover stay parallel with this job.
- name: Publish packages to auto-testing release assets
env:
ASSETS_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
@@ -548,90 +360,6 @@ jobs:
DEB_DATE: ${{ steps.deb.outputs.deb_date }}
BUILD_REF: ${{ env.NIGHTLY_BUILD_REF }}
run: bash scripts/release/publish_nightly_assets.sh
# Cross-repo repository_dispatch cannot use GITHUB_TOKEN. The secret is a
# PAT or GitHub App token for rustfs/auto-testing: classic `repo` scope,
# or fine-grained Contents write (the permission GitHub checks for
# POST /repos/{owner}/{repo}/dispatches). A dispatch failure fails this
# step, and therefore this job, so the scheduled failure alert fires.
# GitHub allows at most 10 top-level client_payload properties, so the
# schema 2 candidate object is nested under `candidate` and the release
# fields sit beside it.
- name: Dispatch nightly candidate to auto-testing
if: ${{ steps.publish.outputs.candidate_file != '' }}
env:
DISPATCH_TOKEN: ${{ secrets.AUTO_TESTING_DISPATCH_TOKEN }}
CANDIDATE_FILE: ${{ steps.publish.outputs.candidate_file }}
DEB_DATE: ${{ steps.deb.outputs.deb_date }}
SOURCE_REPO: ${{ github.repository }}
shell: bash
run: |
set -euo pipefail
: "${DISPATCH_TOKEN:?AUTO_TESTING_DISPATCH_TOKEN is required to dispatch rustfs/auto-testing}"
: "${CANDIDATE_FILE:?nightly candidate manifest is required}"
: "${DEB_DATE:?DEB_DATE is required}"
[[ "${DEB_DATE}" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]]
[[ "${SOURCE_REPO}" == "rustfs/rustfs" ]]
payload="${RUNNER_TEMP}/nightly-candidate-dispatch-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}.json"
jq -n --slurpfile candidate "${CANDIDATE_FILE}" \
--arg event_type "nightly-candidate-ready" \
--arg source_repo "${SOURCE_REPO}" \
--arg latest_alias_url "https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb" \
--arg auto_testing_release "https://github.com/rustfs/auto-testing/releases/tag/nightly" \
--arg channel "nightly" \
--arg deb_date "${DEB_DATE}" \
--arg nightly_workflow ".github/workflows/nightly-gnu.yml" \
'{
event_type: $event_type,
source_repo: $source_repo,
latest_alias_url: $latest_alias_url,
auto_testing_release: $auto_testing_release,
channel: $channel,
deb_date: $deb_date,
nightly_workflow: $nightly_workflow,
candidate: $candidate[0]
}' > "${payload}"
if ! jq -e '
.event_type == "nightly-candidate-ready"
and .source_repo == "rustfs/rustfs"
and .channel == "nightly"
and .deb_date == env.DEB_DATE
and .nightly_workflow == ".github/workflows/nightly-gnu.yml"
and .latest_alias_url == "https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb"
and .auto_testing_release == "https://github.com/rustfs/auto-testing/releases/tag/nightly"
and .candidate.schema == 2
and ((.candidate | keys | sort) == ["build_run_attempt","build_run_id","package_sha256","package_url","schema","source_ref","source_sha","workflow_sha"])
and ((keys | sort) == ["auto_testing_release","candidate","channel","deb_date","event_type","latest_alias_url","nightly_workflow","source_repo"])
' "${payload}" >/dev/null; then
echo "nightly dispatch payload does not match the nightly-candidate-ready contract" >&2
exit 1
fi
body="${RUNNER_TEMP}/nightly-candidate-dispatch-body-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}.json"
jq -n --slurpfile client_payload "${payload}" \
'{event_type: "nightly-candidate-ready", client_payload: $client_payload[0]}' > "${body}"
response="${RUNNER_TEMP}/nightly-candidate-dispatch-response-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}.txt"
status="$(curl -sS --retry 3 --connect-timeout 15 --max-time 60 \
-o "${response}" -w '%{http_code}' \
-X POST \
-H "Authorization: Bearer ${DISPATCH_TOKEN}" \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
-H "Content-Type: application/json" \
--data-binary @"${body}" \
https://api.github.com/repos/rustfs/auto-testing/dispatches)" || {
echo "repository_dispatch request to rustfs/auto-testing failed before an HTTP status" >&2
exit 1
}
if [[ "${status}" != "204" ]]; then
echo "repository_dispatch to rustfs/auto-testing failed: HTTP ${status}" >&2
if [[ -s "${response}" ]]; then
cat "${response}" >&2
fi
exit 1
fi
echo "dispatched nightly-candidate-ready to rustfs/auto-testing (deb_date=${DEB_DATE}, run=${GITHUB_RUN_ID}, attempt=${GITHUB_RUN_ATTEMPT})"
# Live-Vault lane for the rustfs-kms suite (rustfs/backlog#1774).
#
# RUSTFS_KMS_VAULT_TOKEN is the single switch that adds the Vault KV2 and
+2 -2
View File
@@ -47,8 +47,8 @@ permissions:
contents: read
concurrency:
group: oidc-keycloak-live-${{ github.event_name }}-${{ github.ref }}
cancel-in-progress: ${{ github.event_name != 'schedule' }}
group: oidc-keycloak-live-${{ github.ref }}
cancel-in-progress: true
jobs:
oidc-keycloak-live:
+630 -39
View File
@@ -1,53 +1,644 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Build candidate packages
on:
workflow_call:
inputs:
candidate_manifest:
type: string
required: true
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Package Workflow - Build DEB/RPM packages
#
# This workflow builds DEB and RPM packages from pre-built Linux binaries
# and uploads them to Cloudflare R2 and the GitHub release.
#
# Trigger:
# - workflow_run: automatically package after "Build and Release" completes
# for a release tag (the mac/windows/linux binaries are already uploaded
# to the GitHub release before packaging starts)
# - workflow_dispatch: manual fallback with a release tag and/or exact build run ID
#
# Flow:
# 1. Resolve and validate the selected Build workflow run and source identity
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
# aarch64-musl) from build artifacts
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
permissions:
contents: read
# contents: write is required to upload packages to the GitHub release
contents: write
actions: read
on:
# Follows the same pattern as docker.yml: run after the release build
# workflow completes, so packaging is triggered only by release tags
# (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
inputs:
tag:
description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
required: false
type: string
build_run_id:
description: "Build workflow run ID (when combined with tag, both must identify the same release commit)"
required: false
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
cancel-in-progress: true
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
jobs:
# Resolve which build run to use and extract version info
resolve:
name: Resolve Build
# Auto-trigger only from successful tag builds of "Build and Release".
# Tag pushes arrive as event == push with head_branch != main (a
# non-main push head_branch is the release tag name). Manual dispatch
# stays available as a fallback for backfills and re-runs.
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
build_type: ${{ steps.resolve.outputs.build_type }}
build_run_id: ${{ steps.resolve.outputs.build_run_id }}
build_run_number: ${{ steps.resolve.outputs.build_run_number }}
head_sha: ${{ steps.resolve.outputs.head_sha }}
dev_sequence: ${{ steps.resolve.outputs.dev_sequence }}
tag: ${{ steps.resolve.outputs.tag }}
steps:
- name: Resolve build run
id: resolve
shell: bash
env:
GH_TOKEN: ${{ github.token }}
EVENT_NAME: ${{ github.event_name }}
REPOSITORY: ${{ github.repository }}
INPUT_TAG: ${{ github.event.inputs.tag }}
INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
run: |
set -euo pipefail
fail() {
echo "❌ $1" >&2
exit 1
}
TAG=""
BUILD_RUN_ID=""
case "$EVENT_NAME" in
workflow_run)
TAG="$HEAD_BRANCH"
BUILD_RUN_ID="$WORKFLOW_RUN_ID"
;;
workflow_dispatch)
TAG="$INPUT_TAG"
BUILD_RUN_ID="$INPUT_RUN_ID"
;;
*) fail "unsupported event: $EVENT_NAME" ;;
esac
# Validate and classify tags before using them in API paths or logs.
semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
if [[ -n "$TAG" ]]; then
if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
BUILD_TYPE=preview
elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
BUILD_TYPE=prerelease
elif [[ "$TAG" =~ ^${semver_core}$ ]]; then
BUILD_TYPE=release
else
fail "tag is not a supported strict package version"
fi
else
BUILD_TYPE=development
fi
if [[ -n "$BUILD_RUN_ID" ]]; then
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "build run ID must be a positive decimal integer"
echo "Using selected build run: $BUILD_RUN_ID"
elif [[ -n "$TAG" ]]; then
echo "Looking for build run for tag: $TAG"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
if [[ -z "$BUILD_RUN_ID" ]]; then
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f event=push -f status=success -F per_page=100 2>/dev/null |
jq -r --arg tag "$TAG" \
'[.workflow_runs[] | select(.head_branch == $tag)][0].id // empty' || true)
fi
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful build run found for tag"
echo "Found build run: $BUILD_RUN_ID"
else
echo "No tag specified, looking for latest main build"
BUILD_RUN_ID=$(gh api --method GET \
"repos/${REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch=main -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' 2>/dev/null || true)
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ ]] || fail "no successful main build found"
echo "Latest main build: $BUILD_RUN_ID"
fi
# Fetch once and use the same immutable run metadata for identity,
# ordering, workflow provenance, and release-channel validation.
RUN_JSON=$(gh api "repos/${REPOSITORY}/actions/runs/${BUILD_RUN_ID}") ||
fail "cannot read selected build run"
RUN_ID=$(jq -r '.id // empty' <<<"$RUN_JSON")
RUN_NUMBER=$(jq -r '.run_number // empty' <<<"$RUN_JSON")
RUN_STATUS=$(jq -r '.status // empty' <<<"$RUN_JSON")
RUN_CONCLUSION=$(jq -r '.conclusion // empty' <<<"$RUN_JSON")
RUN_PATH=$(jq -r '.path // empty' <<<"$RUN_JSON")
HEAD_SHA=$(jq -r '.head_sha // empty' <<<"$RUN_JSON")
RUN_HEAD_BRANCH=$(jq -r '.head_branch // empty' <<<"$RUN_JSON")
[[ "$RUN_ID" == "$BUILD_RUN_ID" ]] || fail "run metadata ID mismatch"
[[ "$RUN_NUMBER" =~ ^[1-9][0-9]*$ ]] || fail "build run number must be a positive decimal integer"
[[ "$RUN_STATUS" == completed && "$RUN_CONCLUSION" == success ]] || fail "selected build run is not successful"
[[ "$RUN_PATH" == .github/workflows/build.yml ]] || fail "selected run is not Build and Release"
[[ "$HEAD_SHA" =~ ^[0-9a-f]{40}$ ]] || fail "selected build run has an invalid head SHA"
[[ "$RUN_HEAD_BRANCH" != *$'\n'* && -n "$RUN_HEAD_BRANCH" ]] || fail "selected build run has an invalid head branch"
if [[ -n "$TAG" ]]; then
[[ "$RUN_HEAD_BRANCH" == "$TAG" ]] || fail "tag and build run head branch do not match"
TAG_REF_JSON=$(gh api "repos/${REPOSITORY}/git/ref/tags/${TAG}") ||
fail "cannot resolve release tag ref"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_REF_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_REF_JSON")
depth=0
while [[ "$TAG_OBJECT_TYPE" == tag && $depth -lt 5 ]]; do
TAG_OBJECT_JSON=$(gh api "repos/${REPOSITORY}/git/tags/${TAG_OBJECT_SHA}") ||
fail "cannot peel annotated release tag"
TAG_OBJECT_TYPE=$(jq -r '.object.type // empty' <<<"$TAG_OBJECT_JSON")
TAG_OBJECT_SHA=$(jq -r '.object.sha // empty' <<<"$TAG_OBJECT_JSON")
depth=$((depth + 1))
done
[[ "$TAG_OBJECT_TYPE" == commit && "$TAG_OBJECT_SHA" =~ ^[0-9a-f]{40}$ ]] ||
fail "release tag does not resolve to a commit"
[[ "$TAG_OBJECT_SHA" == "$HEAD_SHA" ]] || fail "release tag commit and build run head SHA do not match"
VERSION="$TAG"
DEV_SEQUENCE=""
else
VERSION="dev-${HEAD_SHA}"
DEV_SEQUENCE="$RUN_NUMBER"
fi
{
echo "version=$VERSION"
echo "build_type=$BUILD_TYPE"
echo "build_run_id=$BUILD_RUN_ID"
echo "build_run_number=$RUN_NUMBER"
echo "head_sha=$HEAD_SHA"
echo "dev_sequence=$DEV_SEQUENCE"
echo "tag=${TAG}"
} >> "$GITHUB_OUTPUT"
echo "📊 Resolved:"
echo " Version: $VERSION"
echo " Build type: $BUILD_TYPE"
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Build DEB and RPM packages for each architecture and libc variant
package:
runs-on: ubuntu-latest
timeout-minutes: 45
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: resolve
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
strategy:
fail-fast: false
max-parallel: 1
matrix:
target: [linux-x86_64-musl, linux-aarch64-musl, linux-x86_64-gnu, linux-aarch64-gnu]
include:
- arch: x86_64
libc: gnu
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
# gnu binaries dynamically link glibc; musl binaries are static
# and run without a libc dependency.
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: aarch64
libc: gnu
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: x86_64
libc: musl
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-musl"
deb_depends: ""
rpm_depends: ""
- arch: aarch64
libc: musl
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-musl"
deb_depends: ""
rpm_depends: ""
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(inputs.candidate_manifest).source_sha }}
persist-credentials: false
- name: Reuse completed package set
id: reuse
- name: Normalize package metadata
id: versions
shell: bash
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py reuse --kind packages --target "$TARGET"
- name: Install packaging tools
if: steps.reuse.outputs.reused != 'true'
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
ARCH: ${{ matrix.rpm_arch }}
LIBC: ${{ matrix.libc }}
run: |
sudo apt-get update
sudo apt-get install -y ruby ruby-dev build-essential rpm fakeroot cpio
sudo gem install fpm --version 1.17.0 --no-document
- name: Package accepted executable and verify contents
if: steps.reuse.outputs.reused != 'true'
set -euo pipefail
normalized=$(./scripts/release/package_versions.sh \
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
- name: Download binary artifact from build run
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
run-id: ${{ needs.resolve.outputs.build_run_id }}
github-token: ${{ github.token }}
merge-multiple: true
- name: Extract binary
id: binary
shell: bash
run: |
set -euo pipefail
ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
if [[ -z "$ZIP_FILE" ]]; then
echo "❌ No binary artifact found"
find ./binary-artifact -mindepth 1 -maxdepth 1 -print 2>/dev/null || true
exit 1
fi
echo "Found artifact: $ZIP_FILE"
mkdir -p ./bin
unzip -o "$ZIP_FILE" -d ./bin
if [[ ! -f ./bin/rustfs ]]; then
echo "❌ rustfs binary not found in archive"
exit 1
fi
chmod +x ./bin/rustfs
stat --printf='%n %s bytes\n' ./bin/rustfs
echo "✅ Binary extracted"
- name: Build DEB package
id: deb
shell: bash
env:
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
DEB_ARCH: ${{ matrix.deb_arch }}
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
DEB_DEPENDS: ${{ matrix.deb_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
PKG_DIR="${DEB_FILE%.deb}"
echo "Building DEB: ${DEB_FILE}"
mkdir -p "${PKG_DIR}/DEBIAN"
mkdir -p "${PKG_DIR}/usr/bin"
mkdir -p "${PKG_DIR}/etc/default"
mkdir -p "${PKG_DIR}/lib/systemd/system"
mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
chmod 755 "${PKG_DIR}/usr/bin/rustfs"
cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
cat > "${PKG_DIR}/DEBIAN/control" << EOF
Package: rustfs
Version: ${DEB_VERSION}
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
EOF
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
if [[ -n "$DEB_DEPENDS" ]]; then
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
fi
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage (${LIBC} build)
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
EOF
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
# modifications on upgrade instead of silently overwriting them.
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
/etc/default/rustfs
CONFFILES
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
fakeroot dpkg-deb --build "${PKG_DIR}" "$DEB_FILE"
[[ $(dpkg-deb -f "$DEB_FILE" Package) == rustfs ]]
[[ $(dpkg-deb -f "$DEB_FILE" Version) == "$DEB_VERSION" ]]
[[ $(dpkg-deb -f "$DEB_FILE" Architecture) == "$DEB_ARCH" ]]
dpkg-deb --fsys-tarfile "$DEB_FILE" | tar -tf - | grep -Fx './usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$DEB_FILE"
echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
echo "✅ DEB built: $DEB_FILE"
- name: Build RPM package
id: rpm
shell: bash
env:
RPM_VERSION: ${{ steps.versions.outputs.rpm_version }}
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
RPM_ARCH: ${{ matrix.rpm_arch }}
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
RPM_DEPENDS: ${{ matrix.rpm_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
sudo gem install fpm
./scripts/test_package_versions.sh --require-package-managers
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
depends_args=()
if [[ -n "$RPM_DEPENDS" ]]; then
depends_args+=(--depends "$RPM_DEPENDS")
fi
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
# RustFS Environment Configuration
# See https://docs.rustfs.com for more information
# RUSTFS_ACCESS_KEY=<changeme>
# RUSTFS_SECRET_KEY=<changeme>
# RUSTFS_VOLUMES=""
# RUSTFS_ADDRESS=":9000"
# RUSTFS_CONSOLE_ADDRESS=":9001"
# RUSTFS_CONSOLE_ENABLE=true
# RUSTFS_OBS_LOGGER_LEVEL=error
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
ENVEOF
fpm -s dir -t rpm \
--name rustfs \
--version "$RPM_VERSION" \
--iteration "$RPM_RELEASE" \
--architecture "$RPM_ARCH" \
--package "$RPM_FILE" \
"${depends_args[@]}" \
--description "High-performance distributed object storage (${LIBC} build)" \
--maintainer "RustFS Team <support@rustfs.com>" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
--config-files /etc/default/rustfs \
./bin/rustfs=/usr/bin/rustfs \
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
LICENSE=/usr/share/doc/rustfs/LICENSE \
README.md=/usr/share/doc/rustfs/README.md
if [[ ! -f "$RPM_FILE" ]]; then
echo "❌ RPM build failed"
exit 1
fi
RPM_METADATA=$(rpm -qp --qf '%{NAME}\n%{VERSION}\n%{RELEASE}\n%{ARCH}\n' "$RPM_FILE")
EXPECTED_METADATA=$(printf 'rustfs\n%s\n%s\n%s' "$RPM_VERSION" "$RPM_RELEASE" "$RPM_ARCH")
[[ "$RPM_METADATA" == "$EXPECTED_METADATA" ]]
rpm -qpl "$RPM_FILE" | grep -Fx '/usr/bin/rustfs' >/dev/null
stat --printf='%n %s bytes\n' "$RPM_FILE"
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Upload packages to artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
path: |
*.deb
*.rpm
retention-days: 30
- name: Upload packages to Cloudflare R2
if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "⚠️ R2 credentials missing, skipping upload"
exit 0
fi
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/release"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
echo "📤 Uploading to $R2_PATH"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading: $f"
aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Upload complete"
# Also upload as latest for release/prerelease
if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading latest: $(basename "$f")"
aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Latest packages updated"
fi
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ inputs.candidate_manifest }}
TARGET: ${{ matrix.target }}
run: python3 scripts/release/deliver.py package --target "$TARGET" --output delivery
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.reuse.outputs.reused != 'true'
with:
name: packages-${{ matrix.target }}
path: delivery/*
if-no-files-found: error
compression-level: 0
retention-days: 30
TAG: ${{ needs.resolve.outputs.tag }}
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
shell: bash
run: |
set -euo pipefail
# Upload the packages, then refresh the release checksums so the new
# assets are covered, matching the binary release flow.
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
gh release upload "$TAG" "$f" --clobber
fi
done
CHECKSUM_DIR="$(mktemp -d)"
gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
-D "$CHECKSUM_DIR" --clobber
for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
asset="${spec%%:*}"
checksum_cmd="${spec##*:}"
checksum_file="${CHECKSUM_DIR}/${asset}"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# GitHub stores release asset names with '~' normalized to '.'.
# Current package file names embed the raw tag and contain no
# '~', but older releases may still ship such assets, so the
# checksum entries must reference the name as stored on the
# release.
github_base="${base//\~/.}"
# Remove any stale entry (both naming variants), then append
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
mv "${checksum_file}.tmp2" "$checksum_file"
digest=$("$checksum_cmd" -- "$f" | awk '{print $1}')
printf '%s %s\n' "$digest" "$github_base" >> "$checksum_file"
fi
done
echo "📤 Updating ${asset} for release ${TAG}..."
gh release upload "$TAG" "$checksum_file" --clobber
done
echo "✅ GitHub release assets updated"
# Summary
summary:
name: Summary
needs: [ resolve, package ]
# Every completed Build and Release run triggers this workflow; when
# resolve declined it (not a release tag), there is nothing to summarise.
if: always() && needs.resolve.result != 'skipped'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Print summary
shell: bash
env:
SUMMARY_VERSION: ${{ needs.resolve.outputs.version }}
SUMMARY_BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SUMMARY_BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
SUMMARY_PACKAGE_STATUS: ${{ needs.package.result }}
run: |
{
echo "## 📦 Package Summary"
echo ""
echo "| Item | Value |"
echo "|------|-------|"
echo "| Version | \`${SUMMARY_VERSION}\` |"
echo "| Build Type | ${SUMMARY_BUILD_TYPE} |"
echo "| Build Run | #${SUMMARY_BUILD_RUN_ID} |"
echo "| Package Status | ${SUMMARY_PACKAGE_STATUS} |"
} >> "$GITHUB_STEP_SUMMARY"
-133
View File
@@ -1,133 +0,0 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (Linux)
on:
workflow_dispatch:
inputs:
version:
description: Final product version (no preview suffix)
type: string
required: true
source_sha:
description: Full reviewed source commit
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-candidate-${{ inputs.version }}
cancel-in-progress: false
jobs:
freeze:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 15
outputs:
manifest: ${{ steps.freeze.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Freeze or recover original inputs
id: freeze
env:
GH_TOKEN: ${{ github.token }}
VERSION: ${{ inputs.version }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: python3 scripts/release/pipeline.py prepare --version "$VERSION" --source "$SOURCE_SHA" --directory frozen
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.freeze.outputs.reused != 'true'
with:
name: release-inputs
path: frozen/inputs.json
if-no-files-found: error
retention-days: 30
linux:
needs: freeze
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.freeze.outputs.manifest }}
platforms: linux
upgrade:
needs: [freeze, linux]
runs-on: ubuntu-latest
timeout-minutes: 150
env:
RUSTUP_TOOLCHAIN: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ fromJson(needs.freeze.outputs.manifest).workflow_sha }}
persist-credentials: false
- uses: ./.github/actions/setup
with:
rust-version: ${{ fromJson(needs.freeze.outputs.manifest).toolchain }}
cache-shared-key: release-upgrade-harness-hosted
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Verify archived server against previous releases
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_MANIFEST: ${{ needs.freeze.outputs.manifest }}
run: |
python3 - <<'PYVERIFY'
import sys
from pathlib import Path
sys.path.insert(0, 'scripts/release')
import candidate as c, pipeline, verify
inputs = pipeline.manifest_env()
run = pipeline.current_run('release-candidate.yml')
if any(a['name'] == 'release-upgrade' for a in c.artifact_list(run['id'])):
c.download_artifact(run, 'release-upgrade', 'receipts')
pipeline.verify_evidence(inputs, Path('receipts'), {run['id']: run})
c.emit_output('reused', 'true')
else:
target = 'linux-x86_64-gnu'
c.download_artifact(run, f"candidate-{inputs['candidate_id']}-{target}", 'binary')
verify.upgrade(inputs, Path('binary') / f"rustfs-{target}-v{inputs['version']}.zip", Path('receipts'))
PYVERIFY
id: verify
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.verify.outputs.reused != 'true'
with:
name: release-upgrade
path: receipts/*
if-no-files-found: error
retention-days: 30
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: failure()
with:
name: release-upgrade-failure-${{ github.run_attempt }}
path: receipts/*
retention-days: 7
seal-linux:
needs: [freeze, linux, upgrade]
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Collect trusted receipts and seal Linux bytes
id: seal
env:
GH_TOKEN: ${{ github.token }}
run: |
python3 scripts/release/pipeline.py linux-seal --directory sealed
{
echo '## Linux acceptance'
echo 'Use this run ID and manifest SHA-256 when starting Release candidate (complete).'
sha256sum sealed/linux-manifest.json
echo 'Record cluster, performance, integrity and upgrade acceptance in backlog before continuing.'
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-linux
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
-97
View File
@@ -1,97 +0,0 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Release candidate (complete)
on:
workflow_dispatch:
inputs:
candidate_run_id:
description: Successful Linux candidate run ID
type: string
required: true
linux_manifest_sha256:
description: Exact accepted Linux manifest SHA-256
type: string
required: true
acceptance_report:
description: Backlog report accepting cluster, performance, integrity and upgrade results
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-complete-${{ inputs.candidate_run_id }}
cancel-in-progress: false
jobs:
acceptance:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 30
environment: release-linux-acceptance
outputs:
manifest: ${{ steps.accept.outputs.manifest }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Bind acceptance to the exact Linux manifest
id: accept
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: python3 scripts/release/pipeline.py complete-prepare --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory linux
desktop:
needs: acceptance
uses: ./.github/workflows/build.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
platforms: desktop
packages:
needs: acceptance
uses: ./.github/workflows/package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
images:
needs: acceptance
uses: ./.github/workflows/docker.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
chart:
needs: acceptance
uses: ./.github/workflows/helm-package.yml
with:
candidate_manifest: ${{ needs.acceptance.outputs.manifest }}
seal:
needs: [acceptance, desktop, packages, images, chart]
runs-on: ubuntu-latest
timeout-minutes: 45
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Verify every deliverable and seal the candidate
id: seal
env:
GH_TOKEN: ${{ github.token }}
CANDIDATE_RUN: ${{ inputs.candidate_run_id }}
MANIFEST_SHA: ${{ inputs.linux_manifest_sha256 }}
REPORT: ${{ inputs.acceptance_report }}
run: |
python3 scripts/release/pipeline.py complete-seal --run-id "$CANDIDATE_RUN" --digest "$MANIFEST_SHA" --report "$REPORT" --directory sealed
{
echo '## Final publication approval'
echo 'Review this manifest, then explicitly dispatch Publish accepted release with its SHA-256.'
sha256sum sealed/release-manifest.json
} >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
if: steps.seal.outputs.reused != 'true'
with:
name: release-seal
path: sealed/*
if-no-files-found: error
compression-level: 0
retention-days: 30
-193
View File
@@ -1,193 +0,0 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Publish accepted release
on:
workflow_dispatch:
inputs:
complete_run_id:
description: Successful completed candidate run ID
type: string
required: true
manifest_sha256:
description: Exact approved final manifest SHA-256 (authorizes publication)
type: string
required: true
permissions:
contents: read
actions: read
concurrency:
group: release-publish-${{ inputs.complete_run_id }}
cancel-in-progress: false
jobs:
verify:
if: github.repository == 'rustfs/rustfs' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
timeout-minutes: 45
outputs:
version: ${{ steps.verify.outputs.version }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Recheck approved bytes and trusted certification
id: verify
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
run: python3 scripts/release/publish.py verify --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
version:
needs: verify
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: release-version-${{ needs.verify.outputs.version }}
cancel-in-progress: false
environment: release-publication
permissions:
contents: write
actions: read
packages: write
security-events: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- name: Promote verified version
run: python3 scripts/release/publish.py version --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- name: Publish accepted musl amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-amd64.sarif
category: container-image-musl-amd64
- name: Publish accepted musl arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-musl-arm64.sarif
category: container-image-musl-arm64
- name: Publish accepted GNU amd64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-amd64.sarif
category: container-image-gnu-amd64
- name: Publish accepted GNU arm64 scan
uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4
with:
sarif_file: sealed/scan-gnu-arm64.sarif
category: container-image-gnu-arm64
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-version-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
channels:
needs: [verify, version]
runs-on: ubuntu-latest
timeout-minutes: 90
concurrency:
group: rustfs-release-channels
cancel-in-progress: false
permissions:
contents: write
actions: read
packages: write
env:
GH_TOKEN: ${{ github.token }}
COMPLETE_RUN: ${{ inputs.complete_run_id }}
MANIFEST_SHA: ${{ inputs.manifest_sha256 }}
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: auto
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
AWS_RESPONSE_CHECKSUM_VALIDATION: when_required
AWS_EC2_METADATA_DISABLED: "true"
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
fetch-depth: 0
persist-credentials: false
- name: Install promotion tools
run: |
sudo apt-get update
sudo apt-get install -y skopeo
aws --version
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
username: rustfs
password: ${{ secrets.DOCKERHUB_TOKEN }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: ghcr.io
username: ${{ secrets.GHCR_USERNAME }}
password: ${{ secrets.GHCR_PASSWORD }}
- uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
with:
registry: quay.io
username: ${{ secrets.QUAY_USERNAME }}
password: ${{ secrets.QUAY_PASSWORD }}
- name: Share registry credentials with the OCI copier
run: |
install -m 600 "$HOME/.docker/config.json" "$RUNNER_TEMP/registry-auth.json"
echo "REGISTRY_AUTH_FILE=$RUNNER_TEMP/registry-auth.json" >> "$GITHUB_ENV"
- uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install pinned OSS client
run: |
curl --fail --location --retry 3 --output "$RUNNER_TEMP/ossutil.zip" https://gosspublic.alicdn.com/ossutil/v2/2.1.1/ossutil-2.1.1-linux-amd64.zip
printf '%s %s\n' 60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2 "$RUNNER_TEMP/ossutil.zip" | sha256sum --check -
unzip -q "$RUNNER_TEMP/ossutil.zip" -d "$RUNNER_TEMP/ossutil-extract"
install -m 755 "$RUNNER_TEMP/ossutil-extract/ossutil-2.1.1-linux-amd64/ossutil" "$RUNNER_TEMP/ossutil"
echo "OSSUTIL_BIN=$RUNNER_TEMP/ossutil" >> "$GITHUB_ENV"
- name: Promote verified channels
env:
HELM_TOKEN: ${{ secrets.RUSTFS_HELM_PACKAGE }}
OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
OSS_REGION: cn-beijing
OSS_ENDPOINT: https://oss-cn-beijing.aliyuncs.com
run: python3 scripts/release/publish.py channels --run-id "$COMPLETE_RUN" --digest "$MANIFEST_SHA" --directory sealed
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-channels-receipt-${{ github.run_attempt }}
path: publication.json
if-no-files-found: error
retention-days: 90
@@ -93,12 +93,6 @@ jobs:
printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
} >> "${GITHUB_ENV}"
- name: Checkout environment guard
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
path: rustfs-tools
- name: Checkout auto-testing scripts
timeout-minutes: 5
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -124,13 +118,6 @@ jobs:
echo "auto-testing: ${AUTO_TESTING_SHA}"
df -h /data | tail -1
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-matrix.json
run: python3 rustfs-tools/scripts/functional_environment.py enter --suite fault-tolerance
- name: Cleanup environment (before)
timeout-minutes: 5
if: ${{ inputs.cleanup_before != 'false' }}
@@ -228,24 +215,7 @@ jobs:
if-no-files-found: warn
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && inputs.cleanup_after != 'false' && steps.environment.outcome == 'success' }}
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}"
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-matrix.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 rustfs-tools/scripts/functional_environment.py finish --suite fault-tolerance --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-matrix-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-matrix.json
if-no-files-found: error
./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}" || true
@@ -50,6 +50,10 @@ on:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
repository_dispatch:
# Chain handoff: dispatched when the replication suite finishes, ahead of
# the performance suite.
types: [rustfs-chain-fault-tolerance]
permissions:
contents: read
@@ -76,7 +80,7 @@ jobs:
fault-tolerance-test:
runs-on: smoke-testing
timeout-minutes: 60
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -132,13 +136,6 @@ jobs:
echo "auto-testing: ${AUTO_TESTING_SHA}"
df -h /data | tail -1
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-fault-tolerance.json
run: python3 scripts/functional_environment.py enter --suite fault-tolerance
- name: Cleanup environment (before)
timeout-minutes: 5
if: ${{ inputs.cleanup_before != 'false' }}
@@ -197,21 +194,21 @@ jobs:
jq -e --arg testing_sha "${AUTO_TESTING_SHA}" '
.schema_version == 1 and
.auto_testing_sha == $testing_sha and
.selected_scenarios == ["A", "B", "C", "C2", "D", "E", "F"] and
.selected_scenarios == ["A", "B", "C", "C2", "D", "E"] and
.complete == true and
.counts.expected == 54 and
.counts.seen == 54 and
.counts.expected == 44 and
.counts.seen == 44 and
.counts.harness_errors == 0 and
(.expected_cases | length) == 54 and
(.expected_cases | unique | length) == 54 and
(.seen_cases | length) == 54 and
(.seen_cases | unique | length) == 54 and
(.expected_cases | length) == 44 and
(.expected_cases | unique | length) == 44 and
(.seen_cases | length) == 44 and
(.seen_cases | unique | length) == 44 and
(.seen_cases | sort) == (.expected_cases | sort) and
(.missing_cases | length) == 0 and
(.harness_errors | length) == 0 and
(.cases | length) == 54 and
(.cases | length) == 44 and
([.cases[].case_id] | sort) == (.expected_cases | sort) and
([.cases[].case_id] | unique | length) == 54 and
([.cases[].case_id] | unique | length) == 44 and
all(.cases[]; .verdict == "pass" or .verdict == "known-divergence" or .verdict == "UNEXPECTED") and
.counts.unexpected == ([.cases[] | select(.verdict == "UNEXPECTED")] | length) and
.counts.known_divergence == ([.cases[] | select(.verdict == "known-divergence")] | length)
@@ -308,27 +305,58 @@ jobs:
if-no-files-found: warn
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && inputs.cleanup_after != 'false' && steps.environment.outcome == 'success' }}
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}"
./auto-testing/rustfs-fault-tolerance-test.sh --cleanup -y --log-file "${LOG_FILE}" || true
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
- name: "Continue functional chain (next: Performance)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup. A failed
# handoff retries, then files an alert issue in rustfs/backlog.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-fault-tolerance.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite fault-tolerance --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-fault-tolerance.json
if-no-files-found: error
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -uo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
DISPATCHED=0
for attempt in 1 2 3; do
if gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-performance' \
-F 'client_payload[from_suite]=fault-tolerance'; then
echo "dispatched next suite Performance (attempt ${attempt})"
DISPATCHED=1
break
fi
echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
sleep "${attempt}0"
done
if [ "${DISPATCHED:-0}" -ne 1 ]; then
echo "ERROR: functional chain stalled: could not dispatch Performance after 3 attempts" >&2
TITLE="[functional][chain] stalled after fault-tolerance (run ${GITHUB_RUN_ID})"
BODY_FILE="$(mktemp)"
{
echo "The functional chain could not hand off from **fault-tolerance** to **Performance** after 3 attempts."
echo ""
echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Expected next event: 'rustfs-chain-performance'"
echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
echo "- Recovery: re-dispatch manually with"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
echo " gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-performance'"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
} > "${BODY_FILE}"
gh issue create -R rustfs/backlog --title "${TITLE}" \
--body-file "${BODY_FILE}" --label functional-test \
|| gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
|| echo "could not file the stall alert issue either; check the token" >&2
exit 1
fi
- name: Notify on failure
if: failure()
@@ -363,13 +391,3 @@ jobs:
name: functional-chain-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}/fault-tolerance.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.test.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite fault-tolerance --report "${FUNCTIONAL_ARTIFACTS_DIR}/suite.log"
+20 -65
View File
@@ -13,23 +13,12 @@
# limitations under the License.
# Reusable workflows run from this driver's commit in one Actions run. Each
# selected suite still runs after an earlier failure; the final job checks its plan.
# suite still runs after an earlier suite fails; the final job requires all twelve.
name: RustFS Functional Chain
on:
workflow_dispatch:
inputs:
validation_plan:
description: Registered cadence plan (existing suite frequencies are preserved)
type: choice
options: [nightly, weekly, monthly]
required: false
default: nightly
suites:
description: Comma-separated suite IDs, or empty for the complete registered plan
type: string
required: false
default: ''
build_run_id:
description: Successful nightly or validation package run; empty for package-only build
type: string
@@ -227,25 +216,10 @@ jobs:
outputs:
manifest: ${{ steps.candidate.outputs.manifest }}
performance_ready: ${{ steps.perf_probe.outputs.performance_ready }}
plan: ${{ steps.selection.outputs.plan }}
suites: ${{ steps.selection.outputs.suites }}
optional_suites: ${{ steps.selection.outputs.optional_suites }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Select registered suites
id: selection
env:
VALIDATION_PLAN: ${{ inputs.validation_plan || 'nightly' }}
VALIDATION_SUITES: ${{ inputs.suites }}
run: python3 scripts/validation_registry.py select --output "${RUNNER_TEMP}/suite-plan.json"
- name: Retain selected responsibilities
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-plan-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/suite-plan.json
if-no-files-found: error
- name: Resolve published candidate
id: candidate
env:
@@ -265,7 +239,6 @@ jobs:
if-no-files-found: error
- name: Check shared functional fleet runner before scheduling suites
if: ${{ contains(fromJSON(steps.selection.outputs.environments), 'functional-fleet') }}
# The eleven functional suites run on the smoke-testing fleet; a hard
# requirement. The performance lane runs on pf-testing, which is a
# separate fleet - probe it without failing the whole chain so an
@@ -275,26 +248,15 @@ jobs:
run: python3 scripts/check_functional_runners.py smoke-testing
- name: Probe performance fleet runner
if: ${{ contains(fromJSON(steps.selection.outputs.suites), 'performance') }}
id: perf_probe
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
PERFORMANCE_OPTIONAL: ${{ contains(fromJSON(steps.selection.outputs.optional_suites), 'performance') }}
run: |
# TEMPORARY: the pf-testing environment is broken, so the
# performance lane is disabled until it is repaired. Forcing
# offline keeps the chain on its designed skip path: the lane job
# is skipped, aggregate receives --allow-skipped performance
# (required=false in scheduled-validations.json), and the report
# records the optional-lane skip with its reason.
# Restore the probe below once the environment is healthy again.
# if python3 scripts/check_functional_runners.py pf-testing; then
# echo "performance_ready=online" >> "$GITHUB_OUTPUT"
# else
# echo "performance_ready=offline" >> "$GITHUB_OUTPUT"
# test "$PERFORMANCE_OPTIONAL" = true
# fi
echo "performance_ready=offline" >> "$GITHUB_OUTPUT"
if python3 scripts/check_functional_runners.py pf-testing; then
echo "performance_ready=online" >> "$GITHUB_OUTPUT"
else
echo "performance_ready=offline" >> "$GITHUB_OUTPUT"
fi
validation:
if: ${{ github.event_name == 'workflow_dispatch' && !inputs.validation_runner_inspect_only && inputs.validation_retain_evidence_run_id == '' && inputs.validation_package_sha == '' && (inputs.validation_case != '' || inputs.validation_tools_only || inputs.validation_native_tools || inputs.validation_shared_tools || inputs.validation_manager_tools || inputs.validation_cleanup_run_id != '' || inputs.validation_cleanup_retained_evidence) }}
@@ -618,10 +580,6 @@ jobs:
> "${VALIDATION_ROOT}/shared-fixture-manifest.json"
printf 'VALIDATION_HEAL_SHARED_DIRECTORY=%s\n' "${SHARED_DIRECTORY}" >> "${GITHUB_ENV}"
- name: Reject quarantined suite environment before canonical validation
if: ${{ !inputs.validation_tools_only && inputs.validation_cleanup_run_id == '' && inputs.validation_case != 'environment.inspect' }}
run: python3 scripts/functional_environment.py check --suite heal
- name: Run selected canonical validation case
if: ${{ !inputs.validation_tools_only && inputs.validation_cleanup_run_id == '' && inputs.validation_case != 'environment.inspect' }}
env:
@@ -707,7 +665,7 @@ jobs:
upgrade:
needs: [prepare]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'upgrade') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-upgrade-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -715,7 +673,7 @@ jobs:
s3:
needs: [prepare, upgrade]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 's3') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-s3-compat-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -723,7 +681,7 @@ jobs:
kms:
needs: [prepare, s3]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'kms') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-kms-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -731,7 +689,7 @@ jobs:
tier:
needs: [prepare, kms]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'tier') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-tier-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -739,7 +697,7 @@ jobs:
storage:
needs: [prepare, tier]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'storage') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-storage-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -747,7 +705,7 @@ jobs:
heal:
needs: [prepare, storage]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'heal') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-heal-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -755,7 +713,7 @@ jobs:
pool:
needs: [prepare, heal]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'pool') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-pool-expand-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -763,7 +721,7 @@ jobs:
security:
needs: [prepare, pool]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'security') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-security-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -771,7 +729,7 @@ jobs:
replication:
needs: [prepare, security]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'replication') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-replication-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -779,7 +737,7 @@ jobs:
fault-tolerance:
needs: [prepare, replication]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'fault-tolerance') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-fault-tolerance-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -787,15 +745,15 @@ jobs:
table:
needs: [prepare, fault-tolerance]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'table') }}
if: ${{ always() && needs.prepare.result == 'success' }}
uses: ./.github/workflows/rustfs-table-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
secrets: inherit
performance:
needs: [prepare]
if: ${{ always() && needs.prepare.result == 'success' && contains(fromJSON(needs.prepare.outputs.suites || '[]'), 'performance') && needs.prepare.outputs.performance_ready == 'online' }}
needs: [prepare, table]
if: ${{ always() && needs.prepare.result == 'success' && needs.prepare.outputs.performance_ready == 'online' }}
uses: ./.github/workflows/rustfs-performance-test.yml
with:
chain_manifest: ${{ needs.prepare.outputs.manifest }}
@@ -823,10 +781,8 @@ jobs:
env:
CHAIN_MANIFEST: ${{ needs.prepare.outputs.manifest }}
CHAIN_NEEDS: ${{ toJSON(needs) }}
CHAIN_PLAN: ${{ needs.prepare.outputs.plan }}
run: >-
python3 scripts/functional_chain_evidence.py summarize
--allow-skipped "${{ contains(fromJSON(needs.prepare.outputs.optional_suites || '[]'), 'performance') && needs.prepare.outputs.performance_ready != 'online' && 'performance' || '' }}"
--directory "${RUNNER_TEMP}/chain-evidence"
--output "${RUNNER_TEMP}/chain-report.json"
- name: Retain chain report regardless of test verdict
@@ -843,10 +799,9 @@ jobs:
env:
CHAIN_MANIFEST: ${{ needs.prepare.outputs.manifest }}
CHAIN_NEEDS: ${{ toJSON(needs) }}
CHAIN_PLAN: ${{ needs.prepare.outputs.plan }}
run: >-
python3 scripts/functional_chain_evidence.py aggregate
--allow-skipped "${{ contains(fromJSON(needs.prepare.outputs.optional_suites || '[]'), 'performance') && needs.prepare.outputs.performance_ready != 'online' && 'performance' || '' }}"
--allow-skipped "${{ needs.prepare.outputs.performance_ready == 'online' && '' || 'performance' }}"
--directory "${RUNNER_TEMP}/chain-evidence"
--output "${RUNNER_TEMP}/chain-complete.json"
- name: Upload complete-chain evidence
+62 -41
View File
@@ -29,6 +29,11 @@ on:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
repository_dispatch:
# Chain handoff: dispatched when the storage suite finishes. Heal runs
# exactly once per chain; the pool expansion workflow no longer embeds
# its own heal pass.
types: [rustfs-chain-heal]
permissions:
contents: read
@@ -55,11 +60,10 @@ env:
jobs:
heal-test:
runs-on: smoke-testing
# 90-minute heal run (see the step below) plus setup, report and cleanup.
timeout-minutes: 120
timeout-minutes: 60
# Standalone manual run, or one link of the nightly functional chain
# (storage -> heal -> pool). Pool expansion no longer re-runs heal.
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -105,13 +109,6 @@ jobs:
warp --version || true
df -h /data | tail -1
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-heal.json
run: python3 scripts/functional_environment.py enter --suite heal
- name: Cleanup environment (before)
timeout-minutes: 5
if: ${{ inputs.cleanup_before != 'false' }}
@@ -155,9 +152,10 @@ jobs:
./auto-testing/rustfs_heal_test.sh "${ARGS[@]}" --log-file "${LOG_FILE}"
- name: Run heal test (write -> outage -> heal -> verify)
timeout-minutes: 90
timeout-minutes: 45
id: test
# Collect reports and backlog evidence before the final suite verdict.
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
run: |
./auto-testing/rustfs_heal_test.sh \
@@ -243,8 +241,9 @@ jobs:
RESULT=success
fi
# Preserve harness completion separately; the final verdict checks
# every case after retaining reports and cleanup evidence.
# Product gate: failing steps keep the run green — they are reported
# below and tracked in rustfs/backlog. Only harness/environment
# breakdowns turn the workflow red.
STEPS_TOTAL="$(grep -cE '^\[HEAL-STEP\]' "${LOG_FILE}" 2>/dev/null || true)"
STEPS_FAIL="$(grep -cE '^\[HEAL-STEP\].* FAIL$' "${LOG_FILE}" 2>/dev/null || true)"
STEPS_TOTAL=$(( ${STEPS_TOTAL:-0} + 0 )); STEPS_FAIL=$(( ${STEPS_FAIL:-0} + 0 ))
@@ -282,7 +281,7 @@ jobs:
fi
} | tee "${REPORT_FILE}"
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
# Preserve the harness verdict separately from the final case verdict.
# Red only for harness/environment breakdowns; step failures stay green.
[ "${HARNESS_OK}" = "1" ]
- name: Upload functional report to dashboard
@@ -378,9 +377,8 @@ jobs:
if-no-files-found: error
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && inputs.cleanup_after != 'false' && steps.environment.outcome == 'success' }}
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
set -euo pipefail
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
@@ -398,21 +396,54 @@ jobs:
'
done
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
- name: "Continue functional chain (next: Pool expansion)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup. A failed
# handoff must never pass silently: it retries, then files an alert
# issue in rustfs/backlog so a stalled chain is visible.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-heal.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite heal --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-heal-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-heal.json
if-no-files-found: error
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -uo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
DISPATCHED=0
for attempt in 1 2 3; do
if gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-pool' \
-F 'client_payload[from_suite]=heal'; then
echo "dispatched next suite Pool expansion (attempt ${attempt})"
DISPATCHED=1
break
fi
echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
sleep "${attempt}0"
done
if [ "${DISPATCHED:-0}" -ne 1 ]; then
echo "ERROR: functional chain stalled: could not dispatch Pool expansion after 3 attempts" >&2
TITLE="[functional][chain] stalled after heal (run ${GITHUB_RUN_ID})"
BODY_FILE="$(mktemp)"
{
echo "The functional chain could not hand off from **heal** to **Pool expansion** after 3 attempts."
echo ""
echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Expected next event: 'rustfs-chain-pool'"
echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
echo "- Recovery: re-dispatch manually with"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
echo " gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-pool'"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
} > "${BODY_FILE}"
gh issue create -R rustfs/backlog --title "${TITLE}" \
--body-file "${BODY_FILE}" --label functional-test \
|| gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
|| echo "could not file the stall alert issue either; check the token" >&2
exit 1
fi
- name: Notify on failure
if: failure()
@@ -441,13 +472,3 @@ jobs:
name: functional-chain-heal-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-heal-${{ github.run_id }}-${{ github.run_attempt }}/heal.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.test.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite heal --report "${FUNCTIONAL_ARTIFACTS_DIR}/steps.md"
+59 -38
View File
@@ -29,6 +29,9 @@ on:
description: 'Set RUSTFS_KMS_CONFIG_SECRET (runs KMS-107 config sealing)'
required: false
type: string
repository_dispatch:
# Chain handoff: dispatched when the S3 compatibility suite finishes.
types: [rustfs-chain-kms]
permissions:
contents: read
@@ -53,7 +56,7 @@ jobs:
kms-test:
runs-on: smoke-testing
timeout-minutes: 60
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -97,13 +100,6 @@ jobs:
openssl version
docker --version || true
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-kms.json
run: python3 scripts/functional_environment.py enter --suite kms
- name: Cleanup environment (before)
timeout-minutes: 5
run: |
@@ -135,7 +131,8 @@ jobs:
- name: Run KMS suite
id: test
timeout-minutes: 45
# Collect reports and backlog evidence before the final suite verdict.
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
run: |
set -euo pipefail
@@ -188,8 +185,10 @@ jobs:
RESULT=success
fi
# Report harness completion here; the final verdict checks every
# case result after diagnostics and cleanup have been retained.
# Product gate: failing cases keep the run green — they are reported
# below and tracked in rustfs/backlog. Only harness/environment
# breakdowns (the suite never reached case level, or a wholesale
# failure with zero passes) turn the workflow red.
CASES_TOTAL="$(grep -cE '^\| [A-Z][A-Z0-9]*-[0-9]+ .*\| (PASS|FAIL|UNSUPPORTED|RUNNING) \|' "${CASE_TABLE}" 2>/dev/null || true)"
CASES_FAIL="$(grep -cE '^\| [A-Z][A-Z0-9]*-[0-9]+ .*\| FAIL \|' "${CASE_TABLE}" 2>/dev/null || true)"
CASES_TOTAL=$(( ${CASES_TOTAL:-0} + 0 )); CASES_FAIL=$(( ${CASES_FAIL:-0} + 0 ))
@@ -227,7 +226,7 @@ jobs:
fi
} | tee "${REPORT_FILE}"
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
# Preserve the harness verdict separately from the final case verdict.
# Red only for harness/environment breakdowns; case failures stay green.
[ "${HARNESS_OK}" = "1" ]
- name: Upload functional report to dashboard
@@ -326,9 +325,8 @@ jobs:
if-no-files-found: error
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && steps.environment.outcome == 'success' }}
if: always()
run: |
set -euo pipefail
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
@@ -346,21 +344,54 @@ jobs:
'
done
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
- name: "Continue functional chain (next: Tier)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup. A failed
# handoff must never pass silently: it retries, then files an alert
# issue in rustfs/backlog so a stalled chain is visible.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-kms.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite kms --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-kms-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-kms.json
if-no-files-found: error
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -uo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
DISPATCHED=0
for attempt in 1 2 3; do
if gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-tier' \
-F 'client_payload[from_suite]=kms'; then
echo "dispatched next suite Tier (attempt ${attempt})"
DISPATCHED=1
break
fi
echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
sleep "${attempt}0"
done
if [ "${DISPATCHED:-0}" -ne 1 ]; then
echo "ERROR: functional chain stalled: could not dispatch Tier after 3 attempts" >&2
TITLE="[functional][chain] stalled after kms (run ${GITHUB_RUN_ID})"
BODY_FILE="$(mktemp)"
{
echo "The functional chain could not hand off from **kms** to **Tier** after 3 attempts."
echo ""
echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Expected next event: 'rustfs-chain-tier'"
echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
echo "- Recovery: re-dispatch manually with"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
echo " gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-tier'"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
} > "${BODY_FILE}"
gh issue create -R rustfs/backlog --title "${TITLE}" \
--body-file "${BODY_FILE}" --label functional-test \
|| gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
|| echo "could not file the stall alert issue either; check the token" >&2
exit 1
fi
- name: Notify on failure
if: failure()
@@ -388,13 +419,3 @@ jobs:
name: functional-chain-kms-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-kms-${{ github.run_id }}-${{ github.run_attempt }}/kms.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.test.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite kms --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md"
+6 -37
View File
@@ -54,6 +54,9 @@ on:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
repository_dispatch:
# Chain handoff: dispatched when the replication suite finishes.
types: [rustfs-chain-performance]
permissions:
contents: read
@@ -86,7 +89,7 @@ env:
jobs:
runner-check:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
@@ -110,7 +113,7 @@ jobs:
RUSTFS_WARP_CONCURRENCY: ${{ inputs.warp_concurrency || '64' }}
# Run on manual dispatch, or when the nightly build completed successfully.
# Skipped when nightly failed.
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -156,13 +159,6 @@ jobs:
warp --version || true
df -h /data | tail -1
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-performance.json
run: python3 scripts/functional_environment.py enter --suite performance
- name: Reset test environment (before)
if: ${{ inputs.cleanup_before != 'false' }}
run: |
@@ -327,27 +323,10 @@ jobs:
if-no-files-found: error
- name: Reset test environment (after)
id: environment_cleanup
if: ${{ always() && inputs.cleanup_after != 'false' && steps.environment.outcome == 'success' }}
if: ${{ always() && inputs.cleanup_after != 'false' }}
run: |
./auto-testing/rustfs_performance_test.sh --step 7 -y --log-file "${LOG_FILE:-/dev/null}"
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-performance.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite performance --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-performance-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-performance.json
if-no-files-found: error
- name: Notify on failure
if: failure()
run: |
@@ -375,13 +354,3 @@ jobs:
name: functional-chain-performance-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-performance-${{ github.run_id }}-${{ github.run_attempt }}/performance.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.benchmark.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite performance --report "${RUSTFS_RESULT_DIR}/summary.tsv"
+56 -36
View File
@@ -56,6 +56,9 @@ on:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
repository_dispatch:
# Chain handoff: dispatched when the heal suite finishes.
types: [rustfs-chain-pool]
permissions:
contents: read
@@ -83,13 +86,14 @@ env:
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
jobs:
# Pool expansion owns an independent reusable suite.
# Pool expansion: dispatched by the heal suite's chain handoff. Heal
# itself lives in rustfs-heal-test.yml and runs exactly once per chain.
pool-expansion-test:
name: Pool expansion / decommission test
runs-on: smoke-testing
# Leave setup and finalizer headroom beyond the default 240-minute test.
timeout-minutes: 360
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
env:
RUSTFS_POOL_ADMIN_ENDPOINT: ${{ secrets.RUSTFS_POOL_ADMIN_ENDPOINT || vars.RUSTFS_POOL_ADMIN_ENDPOINT || 'http://rustfs-node1:9000' }}
RUSTFS_POOL_PROXY_ENDPOINT: http://127.0.0.1:19000
@@ -145,13 +149,6 @@ jobs:
exit 1
fi
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-pool.json
run: python3 scripts/functional_environment.py enter --suite pool
- name: Cleanup environment (before)
timeout-minutes: 5
if: ${{ inputs.cleanup_before != 'false' }}
@@ -273,7 +270,8 @@ jobs:
# timeout-minutes requires; the bare || form yielded the string '240'
# and the runner rejected the template (Unexpected value '240').
timeout-minutes: ${{ fromJSON(inputs.pool_timeout_minutes || '240') }}
# Collect reports and backlog evidence before the final suite verdict.
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
run: |
set -o pipefail
@@ -700,9 +698,8 @@ jobs:
./auto-testing/rustfs_pool_nginx_stage.sh cleanup
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && steps.topology.outcome == 'success' && inputs.cleanup_after != 'false' && steps.environment.outcome == 'success' }}
if: ${{ always() && steps.topology.outcome == 'success' && inputs.cleanup_after != 'false' }}
run: |
set -euo pipefail
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
@@ -720,21 +717,54 @@ jobs:
'
done
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
- name: "Continue functional chain (next: Security)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup. A failed
# handoff must never pass silently: it retries, then files an alert
# issue in rustfs/backlog so a stalled chain is visible.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-pool.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite pool --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-pool-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-pool.json
if-no-files-found: error
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -uo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
DISPATCHED=0
for attempt in 1 2 3; do
if gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-security' \
-F 'client_payload[from_suite]=pool'; then
echo "dispatched next suite Security (attempt ${attempt})"
DISPATCHED=1
break
fi
echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
sleep "${attempt}0"
done
if [ "${DISPATCHED:-0}" -ne 1 ]; then
echo "ERROR: functional chain stalled: could not dispatch Security after 3 attempts" >&2
TITLE="[functional][chain] stalled after pool (run ${GITHUB_RUN_ID})"
BODY_FILE="$(mktemp)"
{
echo "The functional chain could not hand off from **pool** to **Security** after 3 attempts."
echo ""
echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Expected next event: 'rustfs-chain-security'"
echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
echo "- Recovery: re-dispatch manually with"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
echo " gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-security'"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
} > "${BODY_FILE}"
gh issue create -R rustfs/backlog --title "${TITLE}" \
--body-file "${BODY_FILE}" --label functional-test \
|| gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
|| echo "could not file the stall alert issue either; check the token" >&2
exit 1
fi
- name: Notify on failure
if: failure()
@@ -763,13 +793,3 @@ jobs:
name: functional-chain-pool-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-pool-${{ github.run_id }}-${{ github.run_attempt }}/pool.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.pool_test.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite pool --report "${POOL_ARTIFACT_DIR}/pool-steps.md"
+55 -38
View File
@@ -39,6 +39,9 @@ on:
- bucket
- site
default: all
repository_dispatch:
# Chain handoff: dispatched when the security suite finishes.
types: [rustfs-chain-replication]
permissions:
contents: read
@@ -65,7 +68,7 @@ jobs:
replication-test:
runs-on: smoke-testing
timeout-minutes: 60
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -110,13 +113,6 @@ jobs:
aws --version
df -h /data | tail -1 || true
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-replication.json
run: python3 scripts/functional_environment.py enter --suite replication
- name: Cleanup environment (before)
timeout-minutes: 5
run: |
@@ -139,7 +135,8 @@ jobs:
- name: Run replication suite
id: test
timeout-minutes: 45
# Collect reports and backlog evidence before the final suite verdict.
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
run: |
set -euo pipefail
@@ -195,8 +192,10 @@ jobs:
RESULT=success
fi
# Report harness completion here; the final verdict checks every
# case result after diagnostics and cleanup have been retained.
# Product gate: failing cases keep the run green — they are reported
# below and tracked in rustfs/backlog. Only harness/environment
# breakdowns (the suite never reached case level, or a wholesale
# failure with zero passes) turn the workflow red.
CASES_TOTAL="$(grep -cE '^\| [A-Z][A-Z0-9]*-[0-9]+ .*\| (PASS|FAIL|UNSUPPORTED|RUNNING) \|' "${CASE_TABLE}" 2>/dev/null || true)"
CASES_FAIL="$(grep -cE '^\| [A-Z][A-Z0-9]*-[0-9]+ .*\| FAIL \|' "${CASE_TABLE}" 2>/dev/null || true)"
CASES_TOTAL=$(( ${CASES_TOTAL:-0} + 0 )); CASES_FAIL=$(( ${CASES_FAIL:-0} + 0 ))
@@ -235,7 +234,7 @@ jobs:
fi
} | tee "${REPORT_FILE}"
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
# Preserve the harness verdict separately from the final case verdict.
# Red only for harness/environment breakdowns; case failures stay green.
[ "${HARNESS_OK}" = "1" ]
- name: Upload functional report to dashboard
@@ -334,9 +333,8 @@ jobs:
if-no-files-found: error
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && steps.environment.outcome == 'success' }}
if: always()
run: |
set -euo pipefail
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
@@ -354,21 +352,50 @@ jobs:
'
done
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
- name: "Continue functional chain (next: Fault tolerance)"
if: ${{ always() && github.event_name == 'repository_dispatch' }}
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-replication.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite replication --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-replication-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-replication.json
if-no-files-found: error
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -uo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
DISPATCHED=0
for attempt in 1 2 3; do
if gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-fault-tolerance' \
-F 'client_payload[from_suite]=replication'; then
echo "dispatched next suite Fault tolerance (attempt ${attempt})"
DISPATCHED=1
break
fi
echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
sleep "${attempt}0"
done
if [ "${DISPATCHED:-0}" -ne 1 ]; then
echo "ERROR: functional chain stalled: could not dispatch Fault tolerance after 3 attempts" >&2
TITLE="[functional][chain] stalled after replication (run ${GITHUB_RUN_ID})"
BODY_FILE="$(mktemp)"
trap 'rm -f "${BODY_FILE}"' EXIT
{
echo "The functional chain could not hand off from **replication** to **Fault tolerance** after 3 attempts."
echo ""
echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Expected next event: 'rustfs-chain-fault-tolerance'"
echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
echo "- Recovery: re-dispatch manually with"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
echo " gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-fault-tolerance'"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
} > "${BODY_FILE}"
gh issue create -R rustfs/backlog --title "${TITLE}" \
--body-file "${BODY_FILE}" --label functional-test \
|| gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
|| echo "could not file the stall alert issue either; check the token" >&2
exit 1
fi
- name: Notify on failure
if: failure()
@@ -397,13 +424,3 @@ jobs:
name: functional-chain-replication-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-replication-${{ github.run_id }}-${{ github.run_attempt }}/replication.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.test.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite replication --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md"
+59 -38
View File
@@ -17,6 +17,9 @@ on:
description: 'Direct .deb URL (nightly/R2/dev). Overrides rustfs_version.'
required: false
type: string
repository_dispatch:
# Chain handoff: dispatched when the upgrade suite finishes.
types: [rustfs-chain-s3]
permissions:
contents: read
@@ -41,7 +44,7 @@ jobs:
s3-compat-test:
runs-on: smoke-testing
timeout-minutes: 60
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -85,13 +88,6 @@ jobs:
openssl version
df -h /data | tail -1
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-s3.json
run: python3 scripts/functional_environment.py enter --suite s3
- name: Cleanup environment (before)
timeout-minutes: 5
run: |
@@ -114,7 +110,8 @@ jobs:
- name: Run S3 compatibility suite
id: test
timeout-minutes: 45
# Collect reports and backlog evidence before the final suite verdict.
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
run: |
set -euo pipefail
@@ -164,8 +161,10 @@ jobs:
RESULT=success
fi
# Report harness completion here; the final verdict checks every
# case result after diagnostics and cleanup have been retained.
# Product gate: failing cases keep the run green — they are reported
# below and tracked in rustfs/backlog. Only harness/environment
# breakdowns (the suite never reached case level, or a wholesale
# failure with zero passes) turn the workflow red.
CASES_TOTAL="$(grep -cE '^\| [A-Z][A-Z0-9]*-[0-9]+ .*\| (PASS|FAIL|UNSUPPORTED|RUNNING) \|' "${CASE_TABLE}" 2>/dev/null || true)"
CASES_FAIL="$(grep -cE '^\| [A-Z][A-Z0-9]*-[0-9]+ .*\| FAIL \|' "${CASE_TABLE}" 2>/dev/null || true)"
CASES_TOTAL=$(( ${CASES_TOTAL:-0} + 0 )); CASES_FAIL=$(( ${CASES_FAIL:-0} + 0 ))
@@ -204,7 +203,7 @@ jobs:
fi
} | tee "${REPORT_FILE}"
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
# Preserve the harness verdict separately from the final case verdict.
# Red only for harness/environment breakdowns; case failures stay green.
[ "${HARNESS_OK}" = "1" ]
- name: Upload functional report to dashboard
@@ -303,9 +302,8 @@ jobs:
if-no-files-found: error
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && steps.environment.outcome == 'success' }}
if: always()
run: |
set -euo pipefail
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
@@ -323,21 +321,54 @@ jobs:
'
done
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
- name: "Continue functional chain (next: KMS)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup. A failed
# handoff must never pass silently: it retries, then files an alert
# issue in rustfs/backlog so a stalled chain is visible.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-s3.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite s3 --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-s3-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-s3.json
if-no-files-found: error
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -uo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
DISPATCHED=0
for attempt in 1 2 3; do
if gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-kms' \
-F 'client_payload[from_suite]=s3'; then
echo "dispatched next suite KMS (attempt ${attempt})"
DISPATCHED=1
break
fi
echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
sleep "${attempt}0"
done
if [ "${DISPATCHED:-0}" -ne 1 ]; then
echo "ERROR: functional chain stalled: could not dispatch KMS after 3 attempts" >&2
TITLE="[functional][chain] stalled after s3 (run ${GITHUB_RUN_ID})"
BODY_FILE="$(mktemp)"
{
echo "The functional chain could not hand off from **s3** to **KMS** after 3 attempts."
echo ""
echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Expected next event: 'rustfs-chain-kms'"
echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
echo "- Recovery: re-dispatch manually with"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
echo " gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-kms'"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
} > "${BODY_FILE}"
gh issue create -R rustfs/backlog --title "${TITLE}" \
--body-file "${BODY_FILE}" --label functional-test \
|| gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
|| echo "could not file the stall alert issue either; check the token" >&2
exit 1
fi
- name: Notify on failure
if: failure()
@@ -365,13 +396,3 @@ jobs:
name: functional-chain-s3-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-s3-${{ github.run_id }}-${{ github.run_attempt }}/s3.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.test.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite s3 --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md"
+27 -38
View File
@@ -52,6 +52,9 @@ on:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
repository_dispatch:
# Chain handoff: dispatched when the pool expansion suite finishes.
types: [rustfs-chain-security]
permissions:
contents: read
@@ -78,7 +81,7 @@ jobs:
security-test:
runs-on: smoke-testing
timeout-minutes: 60
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
# Checkout the repository into its own subdirectory. Checking out at
# the workspace root would wipe the auto-testing clone above (that is
@@ -123,13 +126,6 @@ jobs:
docker --version || true
df -h /data | tail -1
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-security.json
run: python3 rustfs-repo/scripts/functional_environment.py enter --suite security
- name: Cleanup environment (before)
timeout-minutes: 5
if: ${{ inputs.cleanup_before != 'false' || github.event_name != 'workflow_dispatch' }}
@@ -153,7 +149,8 @@ jobs:
- name: Run security suite
id: test
timeout-minutes: 45
# Collect reports and backlog evidence before the final suite verdict.
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
env:
REPORT_FILE: ${{ env.SECURITY_ARTIFACTS_DIR }}/suite-report.md
@@ -197,8 +194,9 @@ jobs:
RESULT=success
fi
# The final verdict checks cases after diagnostics and cleanup.
# The suite log lines carry ANSI
# Product gate: failing cases keep the run green — they are reported
# below and tracked in rustfs/backlog. Only harness/environment
# breakdowns turn the workflow red. The suite log lines carry ANSI
# color escapes both before and after the verdict tag
# (e.g. "\e[1;31m[FAIL]\e[0m STS-105 ..."), so the pattern allows
# any number of escapes on each side of it.
@@ -238,7 +236,7 @@ jobs:
fi
} > "${SECURITY_ARTIFACTS_DIR}/report.md"
cat "${SECURITY_ARTIFACTS_DIR}/report.md" >> "${GITHUB_STEP_SUMMARY}"
# Preserve the harness verdict separately from the final case verdict.
# Red only for harness/environment breakdowns; case failures stay green.
[ "${HARNESS_OK}" = "1" ]
- name: Upload functional report to dashboard
@@ -338,9 +336,8 @@ jobs:
retention-days: 3
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && (inputs.cleanup_after != 'false' || github.event_name != 'workflow_dispatch') && steps.environment.outcome == 'success' }}
if: ${{ always() && (inputs.cleanup_after != 'false' || github.event_name != 'workflow_dispatch') }}
run: |
set -euo pipefail
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
@@ -358,21 +355,23 @@ jobs:
'
done
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
- name: "Continue functional chain (next: Replication)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-security.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 rustfs-repo/scripts/functional_environment.py finish --suite security --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-security-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-security.json
if-no-files-found: error
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -euo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
echo "Dispatching next functional suite: Replication"
gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-replication' \
-F 'client_payload[from_suite]=security'
- name: Notify on failure
if: failure()
@@ -401,13 +400,3 @@ jobs:
name: functional-chain-security-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-security-${{ github.run_id }}-${{ github.run_attempt }}/security.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.test.outcome }}
REPORT_OUTCOME: ${{ steps.report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 rustfs-repo/scripts/functional_chain_evidence.py verify-report --suite security --report "${SECURITY_ARTIFACTS_DIR}/suite-report.md"
+59 -38
View File
@@ -26,6 +26,9 @@ on:
- single-multi
- multi-multi
default: all
repository_dispatch:
# Chain handoff: dispatched when the tier suite finishes.
types: [rustfs-chain-storage]
permissions:
contents: read
@@ -50,7 +53,7 @@ jobs:
storage-test:
runs-on: smoke-testing
timeout-minutes: 60
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -94,13 +97,6 @@ jobs:
openssl version
df -h /data | tail -1
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-storage.json
run: python3 scripts/functional_environment.py enter --suite storage
- name: Cleanup environment (before)
timeout-minutes: 5
run: |
@@ -123,7 +119,8 @@ jobs:
- name: Run storage engine suite
id: test
timeout-minutes: 45
# Collect reports and backlog evidence before the final suite verdict.
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
run: |
set -euo pipefail
@@ -179,8 +176,10 @@ jobs:
RESULT=success
fi
# Report harness completion here; the final verdict checks every
# case result after diagnostics and cleanup have been retained.
# Product gate: failing cases keep the run green — they are reported
# below and tracked in rustfs/backlog. Only harness/environment
# breakdowns (the suite never reached case level, or a wholesale
# failure with zero passes) turn the workflow red.
CASES_TOTAL="$(grep -cE '^\| [A-Z][A-Z0-9]*-[0-9]+ .*\| (PASS|FAIL|UNSUPPORTED|RUNNING) \|' "${CASE_TABLE}" 2>/dev/null || true)"
CASES_FAIL="$(grep -cE '^\| [A-Z][A-Z0-9]*-[0-9]+ .*\| FAIL \|' "${CASE_TABLE}" 2>/dev/null || true)"
CASES_TOTAL=$(( ${CASES_TOTAL:-0} + 0 )); CASES_FAIL=$(( ${CASES_FAIL:-0} + 0 ))
@@ -219,7 +218,7 @@ jobs:
fi
} | tee "${REPORT_FILE}"
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
# Preserve the harness verdict separately from the final case verdict.
# Red only for harness/environment breakdowns; case failures stay green.
[ "${HARNESS_OK}" = "1" ]
- name: Upload functional report to dashboard
@@ -318,9 +317,8 @@ jobs:
if-no-files-found: error
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && steps.environment.outcome == 'success' }}
if: always()
run: |
set -euo pipefail
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
@@ -338,21 +336,54 @@ jobs:
'
done
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
- name: "Continue functional chain (next: Heal)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup. A failed
# handoff must never pass silently: it retries, then files an alert
# issue in rustfs/backlog so a stalled chain is visible.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-storage.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite storage --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-storage-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-storage.json
if-no-files-found: error
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -uo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
DISPATCHED=0
for attempt in 1 2 3; do
if gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-heal' \
-F 'client_payload[from_suite]=storage'; then
echo "dispatched next suite Heal (attempt ${attempt})"
DISPATCHED=1
break
fi
echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
sleep "${attempt}0"
done
if [ "${DISPATCHED:-0}" -ne 1 ]; then
echo "ERROR: functional chain stalled: could not dispatch Heal after 3 attempts" >&2
TITLE="[functional][chain] stalled after storage (run ${GITHUB_RUN_ID})"
BODY_FILE="$(mktemp)"
{
echo "The functional chain could not hand off from **storage** to **Heal** after 3 attempts."
echo ""
echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Expected next event: 'rustfs-chain-heal'"
echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
echo "- Recovery: re-dispatch manually with"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
echo " gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-heal'"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
} > "${BODY_FILE}"
gh issue create -R rustfs/backlog --title "${TITLE}" \
--body-file "${BODY_FILE}" --label functional-test \
|| gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
|| echo "could not file the stall alert issue either; check the token" >&2
exit 1
fi
- name: Notify on failure
if: failure()
@@ -380,13 +411,3 @@ jobs:
name: functional-chain-storage-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-storage-${{ github.run_id }}-${{ github.run_attempt }}/storage.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.test.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite storage --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md"
+25 -40
View File
@@ -89,13 +89,6 @@ jobs:
aws --version
python3 --version
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-table.json
run: python3 scripts/functional_environment.py enter --suite table
- name: Cleanup environment (before)
timeout-minutes: 5
continue-on-error: true
@@ -169,8 +162,10 @@ jobs:
RESULT=success
fi
# Report harness completion here; the final verdict checks every
# case result after diagnostics and cleanup have been retained.
# Product gate: failing cases keep the run green — they are reported
# below and tracked in rustfs/backlog. Only harness/environment
# breakdowns (the suite never reached case level, or a wholesale
# failure with zero passes) turn the workflow red.
CASES_TOTAL="$(grep -cE '^\| [A-Z][A-Z0-9]*-[0-9]+ .*\| (PASS|FAIL|UNSUPPORTED|RUNNING) \|' "${CASE_TABLE}" 2>/dev/null || true)"
CASES_FAIL="$(grep -cE '^\| [A-Z][A-Z0-9]*-[0-9]+ .*\| FAIL \|' "${CASE_TABLE}" 2>/dev/null || true)"
CASES_TOTAL=$(( ${CASES_TOTAL:-0} + 0 )); CASES_FAIL=$(( ${CASES_FAIL:-0} + 0 ))
@@ -208,7 +203,7 @@ jobs:
fi
} | tee "${REPORT_FILE}"
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
# Preserve the harness verdict separately from the final case verdict.
# Red only for harness/environment breakdowns; case failures stay green.
[ "${HARNESS_OK}" = "1" ]
- name: Upload functional report to dashboard
@@ -297,12 +292,11 @@ jobs:
retention-days: 30
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && steps.environment.outcome == 'success' }}
if: always()
continue-on-error: true
run: |
set -euo pipefail
cleanup_status=0
set -uo pipefail
for node in ${RUSTFS_NODES}; do
ssh -o BatchMode=yes -o ConnectTimeout=10 "${RUSTFS_SSH_USER}@${node}" '
set -euo pipefail
@@ -313,25 +307,26 @@ jobs:
fi
for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
${SUDO} rm -f /etc/default/rustfs
' || cleanup_status=$?
' || echo "cleanup failed on ${node} (continuing)"
done
exit "${cleanup_status}"
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
- name: "Continue functional chain (next: Performance)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-table.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite table --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-table-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-table.json
if-no-files-found: error
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -euo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
echo "Dispatching next functional suite: Performance"
gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-performance' \
-F 'client_payload[from_suite]=table'
- name: Notify on failure
if: failure()
@@ -363,13 +358,3 @@ jobs:
name: functional-chain-table-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-table-${{ github.run_id }}-${{ github.run_attempt }}/table.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.test.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite table --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md"
+56 -37
View File
@@ -37,6 +37,9 @@ on:
required: false
default: false
type: boolean
repository_dispatch:
# Chain handoff: dispatched when the KMS suite finishes.
types: [rustfs-chain-tier]
permissions:
contents: read
@@ -65,7 +68,7 @@ jobs:
tier-test:
runs-on: smoke-testing
timeout-minutes: 60
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -213,13 +216,6 @@ jobs:
openssl version
df -h /data | tail -1
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-tier.json
run: python3 scripts/functional_environment.py enter --suite tier
- name: Cleanup environment (before)
timeout-minutes: 5
run: |
@@ -273,7 +269,8 @@ jobs:
- name: Run tier suite
id: test
timeout-minutes: 45
# Collect reports and backlog evidence before the final suite verdict.
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
env:
PACKAGE_URL_INPUT: ${{ inputs.package_url }}
@@ -470,9 +467,8 @@ jobs:
if-no-files-found: error
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && steps.environment.outcome == 'success' }}
if: always()
run: |
set -euo pipefail
sudo docker rm -f rustfs-test-mqtt >/dev/null 2>&1 || true
@@ -492,22 +488,6 @@ jobs:
'
done
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-tier.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite tier --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-tier-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-tier.json
if-no-files-found: error
- name: Cleanup pinned RustFS CLI
timeout-minutes: 2
if: always()
@@ -684,6 +664,55 @@ jobs:
--package-source "${PACKAGE_SOURCE}" \
--date "$(date -u +%Y-%m-%d)"
- name: "Continue functional chain (next: Storage engine)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup. A failed
# handoff must never pass silently: it retries, then files an alert
# issue in rustfs/backlog so a stalled chain is visible.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -uo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
DISPATCHED=0
for attempt in 1 2 3; do
if gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-storage' \
-F 'client_payload[from_suite]=tier'; then
echo "dispatched next suite Storage engine (attempt ${attempt})"
DISPATCHED=1
break
fi
echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
sleep "${attempt}0"
done
if [ "${DISPATCHED:-0}" -ne 1 ]; then
echo "ERROR: functional chain stalled: could not dispatch Storage engine after 3 attempts" >&2
TITLE="[functional][chain] stalled after tier (run ${GITHUB_RUN_ID})"
BODY_FILE="$(mktemp)"
{
echo "The functional chain could not hand off from **tier** to **Storage engine** after 3 attempts."
echo ""
echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Expected next event: 'rustfs-chain-storage'"
echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
echo "- Recovery: re-dispatch manually with"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
echo " gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-storage'"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
} > "${BODY_FILE}"
gh issue create -R rustfs/backlog --title "${TITLE}" \
--body-file "${BODY_FILE}" --label functional-test \
|| gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
|| echo "could not file the stall alert issue either; check the token" >&2
exit 1
fi
- name: Notify on failure
if: failure()
run: |
@@ -710,13 +739,3 @@ jobs:
name: functional-chain-tier-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-tier-${{ github.run_id }}-${{ github.run_attempt }}/tier.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.test.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite tier --report "${TIER_ARTIFACTS_DIR}/rustfs-tier-cases.md"
+59 -38
View File
@@ -59,6 +59,9 @@ on:
description: 'Reset the nodes after the test (DESTROYS test data/config)'
type: boolean
default: true
repository_dispatch:
# Functional-chain entry: dispatched by rustfs-functional-chain.yml.
types: [rustfs-chain-upgrade]
permissions:
contents: read
@@ -83,7 +86,7 @@ jobs:
upgrade-test:
runs-on: smoke-testing
timeout-minutes: 60
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' }}
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -129,13 +132,6 @@ jobs:
docker --version || true
df -h /data | tail -1
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-upgrade.json
run: python3 scripts/functional_environment.py enter --suite upgrade
- name: Cleanup environment (before)
timeout-minutes: 5
if: ${{ inputs.cleanup_before != 'false' || github.event_name != 'workflow_dispatch' }}
@@ -169,7 +165,8 @@ jobs:
- name: Run upgrade compatibility suite
id: test
timeout-minutes: 45
# Collect reports and backlog evidence before the final suite verdict.
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
@@ -261,8 +258,10 @@ jobs:
RESULT=success
fi
# Report harness completion here; the final verdict checks every
# case result after diagnostics and cleanup have been retained.
# Product gate: failing cases keep the run green — they are reported
# below and tracked in rustfs/backlog. Only harness/environment
# breakdowns (the suite never reached case level, or a wholesale
# failure with zero passes) turn the workflow red.
CASES_TOTAL="$(grep -cE '^\| [A-Z][A-Z0-9]*-[0-9]+ .*\| (PASS|FAIL|UNSUPPORTED|RUNNING) \|' "${CASE_TABLE}" 2>/dev/null || true)"
CASES_FAIL="$(grep -cE '^\| [A-Z][A-Z0-9]*-[0-9]+ .*\| FAIL \|' "${CASE_TABLE}" 2>/dev/null || true)"
CASES_TOTAL=$(( ${CASES_TOTAL:-0} + 0 )); CASES_FAIL=$(( ${CASES_FAIL:-0} + 0 ))
@@ -305,7 +304,7 @@ jobs:
fi
} | tee "${REPORT_FILE}"
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
# Preserve the harness verdict separately from the final case verdict.
# Red only for harness/environment breakdowns; case failures stay green.
[ "${HARNESS_OK}" = "1" ]
- name: Upload functional report to dashboard
@@ -413,9 +412,8 @@ jobs:
retention-days: 3
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && (inputs.cleanup_after != 'false' || github.event_name != 'workflow_dispatch') && steps.environment.outcome == 'success' }}
if: ${{ always() && (inputs.cleanup_after != 'false' || github.event_name != 'workflow_dispatch') }}
run: |
set -euo pipefail
read -r -a NODES <<< "${RUSTFS_NODES:-vm000 vm001 vm002}"
@@ -433,21 +431,54 @@ jobs:
'
done
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
- name: "Continue functional chain (next: S3 compatibility)"
# Only chain-triggered runs forward to the next suite; standalone
# workflow_dispatch runs stop after their own cleanup. A failed
# handoff must never pass silently: it retries, then files an alert
# issue in rustfs/backlog so a stalled chain is visible.
if: ${{ always() && github.event_name == 'repository_dispatch' }}
continue-on-error: true
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-upgrade.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite upgrade --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Retain environment ownership and cleanup receipt
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-environment-upgrade-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/environment-upgrade.json
if-no-files-found: error
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
run: |
set -uo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; cannot dispatch the next suite" >&2
exit 1
fi
DISPATCHED=0
for attempt in 1 2 3; do
if gh api --method POST repos/rustfs/rustfs/dispatches \
-f event_type='rustfs-chain-s3' \
-F 'client_payload[from_suite]=upgrade'; then
echo "dispatched next suite S3 compatibility (attempt ${attempt})"
DISPATCHED=1
break
fi
echo "dispatch attempt ${attempt} failed; retrying in ${attempt}0s" >&2
sleep "${attempt}0"
done
if [ "${DISPATCHED:-0}" -ne 1 ]; then
echo "ERROR: functional chain stalled: could not dispatch S3 compatibility after 3 attempts" >&2
TITLE="[functional][chain] stalled after upgrade (run ${GITHUB_RUN_ID})"
BODY_FILE="$(mktemp)"
{
echo "The functional chain could not hand off from **upgrade** to **S3 compatibility** after 3 attempts."
echo ""
echo "- Failed suite job: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
echo "- Expected next event: 'rustfs-chain-s3'"
echo "- Likely cause: PF_TESTING_GH_TOKEN lacks contents:write on rustfs/rustfs, or the GitHub API was unavailable."
echo "- Recovery: re-dispatch manually with"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
echo " gh api --method POST repos/rustfs/rustfs/dispatches -f event_type='rustfs-chain-s3'"
FENCE="$(printf "\x60\x60\x60")"; echo " ${FENCE}"
} > "${BODY_FILE}"
gh issue create -R rustfs/backlog --title "${TITLE}" \
--body-file "${BODY_FILE}" --label functional-test \
|| gh issue create -R rustfs/backlog --title "${TITLE}" --body-file "${BODY_FILE}" \
|| echo "could not file the stall alert issue either; check the token" >&2
exit 1
fi
- name: Notify on failure
if: failure()
@@ -477,13 +508,3 @@ jobs:
name: functional-chain-upgrade-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-upgrade-${{ github.run_id }}-${{ github.run_attempt }}/upgrade.json
if-no-files-found: error
- name: Enforce suite and report result
if: ${{ always() }}
env:
SUITE_OUTCOME: ${{ steps.test.outcome }}
REPORT_OUTCOME: ${{ steps.chain_report.outcome }}
run: |
test "$SUITE_OUTCOME" = success
test "$REPORT_OUTCOME" = success
python3 scripts/functional_chain_evidence.py verify-report --suite upgrade --report "${FUNCTIONAL_ARTIFACTS_DIR}/cases.md"
@@ -1,457 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: RustFS Validation Sweep
# Executes the migrated canonical validation cases (rustfs/auto-testing#113)
# on the shared functional fleet. Everything with a lane executor is swept in
# rotating nightly batches so the whole set is covered roughly weekly; a
# single case or one module can be dispatched manually. That is the
# implemented catalog (66 today) plus the eight planned cases whose dedicated
# executors already exist and have live PASS evidence - fresh nightly
# evidence feeds their acceptance review. heal.* cases stay on the
# interactive validation flow: they require reviewed native/tool collectors
# built by separate validation-tools runs before fleet execution.
on:
schedule:
# After the nightly functional chain settles (chain starts ~16:48 UTC and
# runs for about two hours).
- cron: "30 19 * * *"
workflow_dispatch:
inputs:
case:
description: 'Single canonical validation case id (overrides module and rotation), e.g. pool.decommission'
type: string
required: false
default: ''
module:
description: 'Restrict the sweep to one catalog module, e.g. bucket-replication'
type: string
required: false
default: ''
limit:
description: 'Maximum cases per sweep (0 = run everything selected)'
type: number
required: false
default: 12
build_run_id:
description: 'Successful nightly package run; empty resolves the latest'
type: string
required: false
default: ''
build_run_attempt:
description: 'Attempt of that package run'
type: string
required: false
default: '1'
permissions:
contents: read
concurrency:
group: rustfs-validation-sweep
cancel-in-progress: false
env:
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES || 'vm000 vm001 vm002 vm003' }}
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER || 'azureuser' }}
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
jobs:
sweep:
if: github.repository == 'rustfs/rustfs'
runs-on: smoke-testing
timeout-minutes: 150
steps:
- name: Checkout chain tooling
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Resolve nightly candidate
id: candidate
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
BUILD_RUN_ATTEMPT: ${{ inputs.build_run_attempt }}
CHAIN_OUTPUT: ${{ runner.temp }}/validation-sweep-candidate.json
run: |
set -euo pipefail
# Scheduled runs carry no inputs and the resolver only derives a
# build from a workflow_run payload (which a schedule cannot
# provide), so resolve the newest successful nightly build
# explicitly before invoking it.
if [ -z "${BUILD_RUN_ID:-}" ]; then
LATEST="$(gh api "repos/rustfs/rustfs/actions/workflows/nightly-gnu.yml/runs?branch=main&status=success&per_page=1" --jq '.workflow_runs[0]')"
BUILD_RUN_ID="$(printf '%s' "${LATEST}" | jq -r '.id')"
BUILD_RUN_ATTEMPT="$(printf '%s' "${LATEST}" | jq -r '.run_attempt')"
export BUILD_RUN_ID BUILD_RUN_ATTEMPT
echo "scheduled sweep pinned to nightly ${BUILD_RUN_ID} attempt ${BUILD_RUN_ATTEMPT}"
fi
python3 scripts/resolve_functional_candidate.py
python3 - <<'PY'
import json
import os
from pathlib import Path
root = Path(os.environ["RUNNER_TEMP"]) / "validation-sweep"
root.mkdir(parents=True, exist_ok=True)
chain = json.loads((Path(os.environ["RUNNER_TEMP"]) / "validation-sweep-candidate.json").read_text())
manifest = chain["candidate"]["manifest"]
(root / "candidate.json").write_text(json.dumps({
"package_url": manifest["package_url"],
"package_sha256": manifest["package_sha256"],
"source_sha": manifest["source_sha"],
}, sort_keys=True) + "\n")
with open(os.environ["GITHUB_OUTPUT"], "a") as output:
output.write("testing_sha=" + chain["testing_sha"] + "\n")
output.write("package_url=" + manifest["package_url"] + "\n")
output.write("source_sha=" + manifest["source_sha"] + "\n")
PY
- name: Checkout auto-testing scripts
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.candidate.outputs.testing_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
path: auto-testing
persist-credentials: false
- name: Prepare validation Python runtime
# The pool / ilm / replication validation runners require a managed
# Python, exactly like the chain's validation job provides.
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install validation Python requirements
# Case scripts call python3 and import boto3 at startup. Install the
# functional chain's client into a per-run env so that interpreter
# sees it without changing the runner's shared Python.
run: |
set -euo pipefail
validation_python="${RUNNER_TEMP}/rustfs-validation-python-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}"
test ! -e "${validation_python}"
python3 -m venv "${validation_python}"
"${validation_python}/bin/python" -m pip install \
-r auto-testing/scripts/validation_ilm/requirements.txt
printf '%s\n' "${validation_python}/bin" >> "${GITHUB_PATH}"
- name: Select sweep cases
id: select
env:
SELECTED_CASE: ${{ inputs.case }}
SELECTED_MODULE: ${{ inputs.module }}
CASE_LIMIT: ${{ inputs.limit }}
SWEEP_EVENT: ${{ github.event_name }}
SWEEP_ROOT: ${{ runner.temp }}/validation-sweep
run: |
set -euo pipefail
python3 - <<'PY'
import datetime
import json
import os
from pathlib import Path
sweep_root = Path(os.environ["SWEEP_ROOT"])
catalog = json.loads(Path("auto-testing/validation-cases.json").read_text())
def sweepable(case):
status = case.get("source_status")
if status == "implemented":
return True
# The remaining planned non-heal cases route to the lane
# executors and already have live PASS evidence; sweeping them
# nightly produces fresh evidence for their acceptance review.
return status == "planned" and not case["id"].startswith("heal.")
selected = [(case["id"], module["id"]) for module in catalog["modules"]
for case in module["cases"]
if sweepable(case)]
if not selected:
raise SystemExit("catalog has no sweepable cases")
one_case = os.environ.get("SELECTED_CASE", "").strip()
module_filter = os.environ.get("SELECTED_MODULE", "").strip()
known = {case_id for case_id, _ in selected}
modules = {module_id for _, module_id in selected}
if one_case:
if one_case not in known:
raise SystemExit(f"unknown case or no lane executor for it: {one_case}")
selected = [(one_case, dict(selected)[one_case])]
elif module_filter:
if module_filter not in modules:
raise SystemExit(f"unknown module or no implemented cases: {module_filter}")
selected = [pair for pair in selected if pair[1] == module_filter]
elif os.environ.get("SWEEP_EVENT") == "schedule":
# Stateless rotation: every night take one window of the catalog
# so a full sweep needs len(selected)/limit nights and no state
# has to be carried between runs.
limit = max(1, int(os.environ.get("CASE_LIMIT") or 12))
day = datetime.datetime.now(datetime.timezone.utc).timetuple().tm_yday
start = (day * limit) % len(selected)
selected = (selected[start:] + selected[:start])[:limit]
limit = int(os.environ.get("CASE_LIMIT") or 0)
if one_case or module_filter or (not os.environ.get("SWEEP_EVENT") == "schedule"):
if limit > 0:
selected = selected[:limit]
(sweep_root / "sweep-cases.txt").write_text("".join(f"{case_id}\n" for case_id, _ in selected))
with open(os.environ["GITHUB_OUTPUT"], "a") as output:
output.write(f"count={len(selected)}\n")
print(f"selected {len(selected)} case(s):", ", ".join(c for c, _ in selected))
PY
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-validation-sweep.json
run: python3 scripts/functional_environment.py enter --suite heal
- name: Run validation cases
id: run_cases
timeout-minutes: 120
env:
AUTO_TESTING_SHA: ${{ steps.candidate.outputs.testing_sha }}
SWEEP_ROOT: ${{ runner.temp }}/validation-sweep
run: |
set -uo pipefail
mkdir -p "${SWEEP_ROOT}/results"
: > "${SWEEP_ROOT}/results.txt"
while read -r CASE_ID; do
[ -n "${CASE_ID}" ] || continue
echo "::group::case ${CASE_ID}"
rm -rf "${SWEEP_ROOT}/run"
if ./auto-testing/scripts/run_validation_case.sh \
"${CASE_ID}" "${SWEEP_ROOT}/candidate.json" \
"${AUTO_TESTING_SHA}" "${SWEEP_ROOT}/run" \
> "${SWEEP_ROOT}/results/${CASE_ID//\//_}.log" 2>&1; then
printf '%s\tPASS\n' "${CASE_ID}" >> "${SWEEP_ROOT}/results.txt"
echo "case ${CASE_ID}: PASS"
else
printf '%s\tFAIL\n' "${CASE_ID}" >> "${SWEEP_ROOT}/results.txt"
echo "case ${CASE_ID}: FAIL (log retained in the sweep artifact)"
fi
if [ -f "${SWEEP_ROOT}/run/report.md" ]; then
cp "${SWEEP_ROOT}/run/report.md" "${SWEEP_ROOT}/results/${CASE_ID//\//_}.report.md" 2>/dev/null || true
fi
echo "::endgroup::"
# Fleet hygiene between cases: the case runners set up their own
# topology, this purge guarantees the next case starts clean even
# when the previous one died mid-run.
for node in ${RUSTFS_NODES}; do
# -n keeps ssh from consuming the case list this loop reads from stdin
ssh -n -o BatchMode=yes -o ConnectTimeout=10 "${RUSTFS_SSH_USER}@${node}" '
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
${SUDO} systemctl stop rustfs 2>/dev/null || true
if ${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
${SUDO} dpkg -P rustfs
fi
for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
${SUDO} rm -f /etc/default/rustfs
' || echo "inter-case cleanup failed on ${node} (continuing)"
done
done < "${SWEEP_ROOT}/sweep-cases.txt"
echo "sweep finished: $(wc -l < "${SWEEP_ROOT}/results.txt") case(s) executed"
- name: Generate sweep report
if: ${{ always() && steps.environment.outcome == 'success' }}
env:
SWEEP_ROOT: ${{ runner.temp }}/validation-sweep
PACKAGE_URL: ${{ steps.candidate.outputs.package_url }}
SOURCE_SHA: ${{ steps.candidate.outputs.source_sha }}
AUTO_TESTING_SHA: ${{ steps.candidate.outputs.testing_sha }}
run: |
set -euo pipefail
python3 - <<'PY'
import json
import os
from pathlib import Path
root = Path(os.environ["SWEEP_ROOT"])
catalog = json.loads(Path("auto-testing/validation-cases.json").read_text())
module_of = {case["id"]: module["id"] for module in catalog["modules"] for case in module["cases"]}
rows = []
for line in (root / "results.txt").read_text().splitlines():
if not line:
continue
case_id, status = line.split("\t")
rows.append((case_id, module_of.get(case_id, "?"), status))
passed = sum(1 for _, _, s in rows if s == "PASS")
failed = len(rows) - passed
run_url = (f"{os.environ.get('GITHUB_SERVER_URL', 'https://github.com')}/"
f"{os.environ.get('GITHUB_REPOSITORY', 'rustfs/rustfs')}/actions/runs/"
f"{os.environ.get('GITHUB_RUN_ID', '')}")
report = [
"# RustFS validation sweep report", "",
f"- Run: {run_url}",
f"- Attempt: {os.environ.get('GITHUB_RUN_ATTEMPT', '')}",
f"- Trigger: {os.environ.get('GITHUB_EVENT_NAME', '')}",
f"- Package: {os.environ.get('PACKAGE_URL', 'nightly (R2 latest)')}",
f"- Source SHA: {os.environ.get('SOURCE_SHA', '')}",
f"- auto-testing: {os.environ.get('AUTO_TESTING_SHA', '')}", "",
"Canonical validation cases (rustfs/auto-testing#113) executed by the sweep, grouped by catalog module.", "",
]
# Group the executed cases by catalog module (the major category:
# pool lifecycle, heal, ILM expiration, bucket/site replication, kms,
# data services, identity, s3 compatibility). Per-module subtotals
# deliberately use "- Subtotal:" so the dashboard overview regexes
# (which anchor on "- Total:") keep reading the grand totals below.
by_module = {}
module_order = []
for case_id, module, status in rows:
if module not in by_module:
by_module[module] = []
module_order.append(module)
by_module[module].append((case_id, status))
report += ["## Case Summary", ""]
for module in module_order:
cases = by_module[module]
mod_pass = sum(1 for _, st in cases if st == "PASS")
mod_fail = len(cases) - mod_pass
report += [f"### {module} ({len(cases)})", "",
"| Case | Status |", "| --- | --- |"]
report += [f"| {case_id} | {status} |" for case_id, status in cases]
report += ["", f"- Subtotal: {mod_pass} passed, {mod_fail} failed", ""]
report += [
"## Totals", "",
f"- Total: {len(rows)}",
f"- PASS: {passed}",
f"- FAIL: {failed}",
"- UNSUPPORTED: 0", "",
"Failing cases keep this workflow red only as a report; the sweep artifact",
"retains every case log and per-case report for diagnosis.", "",
]
(root / "report.md").write_text("\n".join(report))
(root / "cases.md").write_text("\n".join(f"| {c} | {m} | {s} |" for c, m, s in rows) + "\n")
PY
cat "${SWEEP_ROOT}/report.md" >> "${GITHUB_STEP_SUMMARY}"
- name: Upload functional report to dashboard
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 2
continue-on-error: true
env:
GH_TOKEN: ${{ env.PF_TESTING_GH_TOKEN }}
REPORT_FILE: ${{ runner.temp }}/validation-sweep/report.md
SUITE: validation
run: |
set -euo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; skipping dashboard upload"
exit 0
fi
DATE="$(date -u +%Y-%m-%d)"
REPORT_PATH="functional-reports/${SUITE}/${DATE}.md"
B64_FILE="$(mktemp)"
python3 -c 'import base64,sys;print(base64.b64encode(open(sys.argv[1],"rb").read()).decode())' "${REPORT_FILE}" > "${B64_FILE}"
SHA="$(gh api "repos/rustfs/dashboard/contents/${REPORT_PATH}" -q '.sha' 2>/dev/null || true)"
if [ -n "${SHA}" ]; then
jq -n --arg msg "report(${SUITE}): ${DATE}" --rawfile content "${B64_FILE}" --arg sha "${SHA}" \
'{message:$msg, content:($content|rtrimstr("\n")), sha:$sha}' \
| gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
else
jq -n --arg msg "report(${SUITE}): ${DATE}" --rawfile content "${B64_FILE}" \
'{message:$msg, content:($content|rtrimstr("\n"))}' \
| gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
fi
echo "dashboard updated: ${REPORT_PATH}"
- name: Retain sweep evidence
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-validation-sweep-${{ github.run_id }}-${{ github.run_attempt }}
path: |
${{ runner.temp }}/validation-sweep/report.md
${{ runner.temp }}/validation-sweep/results.txt
${{ runner.temp }}/validation-sweep/sweep-cases.txt
${{ runner.temp }}/validation-sweep/candidate.json
${{ runner.temp }}/validation-sweep/results/
${{ runner.temp }}/validation-sweep/run/
# warn (not error): an early infra failure legitimately leaves no
# per-case evidence, and the failing step is already red.
if-no-files-found: warn
retention-days: 30
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && steps.environment.outcome == 'success' }}
run: |
set -uo pipefail
cleanup_status=0
for node in ${RUSTFS_NODES}; do
ssh -o BatchMode=yes -o ConnectTimeout=10 "${RUSTFS_SSH_USER}@${node}" '
set -euo pipefail
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
${SUDO} systemctl stop rustfs 2>/dev/null || true
if ${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
${SUDO} dpkg -P rustfs
fi
for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
${SUDO} rm -f /etc/default/rustfs
' || cleanup_status=$?
done
exit "${cleanup_status}"
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-validation-sweep.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite heal --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Notify on failure
if: failure()
run: echo "RustFS validation sweep failed (harness/environment breakdown); see the sweep artifact for case logs."
- name: Enforce sweep result
if: ${{ always() && steps.environment.outcome == 'success' }}
env:
SWEEP_ROOT: ${{ runner.temp }}/validation-sweep
run: |
set -euo pipefail
if grep -q $'\tFAIL$' "${SWEEP_ROOT}/results.txt"; then
echo "validation sweep has failing cases:"
grep $'\tFAIL$' "${SWEEP_ROOT}/results.txt"
exit 1
fi
alert-on-failure:
name: Alert on scheduled failure
# A scheduled sweep failure usually means a canonical validation case
# regressed or the fleet environment broke; surface it as a tracking
# issue instead of leaving it in the Actions tab.
needs: [sweep]
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v5.0.0
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}
@@ -34,7 +34,7 @@ jobs:
# self-hosted sm-standard-* pool is down, which is exactly when the
# scheduled validations it watches stall.
runs-on: ubuntu-latest
timeout-minutes: 20
timeout-minutes: 10
permissions:
actions: read
contents: read
@@ -47,7 +47,6 @@ jobs:
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
RUSTFS_DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
NIGHTLY_SOURCE_REF: ${{ vars.NIGHTLY_BRANCH || 'main' }}
run: |
set +e
python3 scripts/check_scheduled_validation_freshness.py \
@@ -16,8 +16,19 @@ name: Scheduled Validation Watchdog
on:
workflow_run:
# The registry filters this broad subscription; names are not duplicated here.
workflows: ["*"]
workflows:
- "Security Audit"
- "Build and Release"
- "Continuous Integration"
- "coverage"
- "e2e-nightly"
- "e2e-distributed"
- "e2e-s3tests"
- "Fuzz"
- "mint"
- "minio-interop"
- "Nightly GNU Build"
- "Runner Hygiene"
types: [completed]
permissions:
@@ -26,13 +37,11 @@ permissions:
jobs:
alert-on-incomplete-run:
name: Alert on incomplete scheduled run
# The registry still owns responsibility and cadence selection. Successful
# and PR runs can never produce an incomplete scheduled-run alert.
if: >-
github.repository == 'rustfs/rustfs' &&
github.event.workflow_run.event == 'schedule' &&
github.event.workflow_run.conclusion != 'success' &&
(github.event.workflow_run.event == 'schedule' ||
github.event.workflow_run.event == 'workflow_run')
github.event.workflow_run.conclusion != 'failure'
# Stay on GitHub-hosted runners: this monitor must keep running when the
# self-hosted sm-standard-* pool is down, which is exactly when the
# scheduled validations it watches stall.
@@ -46,11 +55,7 @@ jobs:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Select expected completion from registry
id: registry
run: python3 scripts/validation_registry.py watchdog
- name: Open or update incomplete-run issue
if: steps.registry.outputs.selected == 'true'
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}
-26
View File
@@ -107,32 +107,6 @@ jobs:
shell: pwsh
run: cargo test -p rustfs-ecstore --lib multipart_owned_record_parent_barriers_ -- --nocapture
# Native multipart completion publishes its payload from a different
# parent than its metadata, and abort renames under a precondition; both
# need their own Windows handle discipline.
- name: Test native multipart complete and abort publication
shell: pwsh
run: >-
cargo test -p rustfs-ecstore --lib --
native_abort_missing_state_with_actual_owned_records_never_becomes_legacy_or_missing
native_abort_pending_intent_is_not_a_delete_certificate_and_raw_cleanup_preserves_exact_evidence
--nocapture
# Owned-undo reclaim removes a directory it pinned, copy heal persists
# fault bytes through a handle, and metadata reads meet a directory.
- name: Test native multipart reclaim, copy heal, and directory reads
shell: pwsh
run: >-
cargo test -p rustfs-ecstore --lib --
native_multipart_owned_undo
native_multipart_copy_heal
orphan_multipart_cleanup_rejects_metadata_directory_on_one_disk
--nocapture
- name: Test directory removal fallback
shell: pwsh
run: "cargo test -p rustfs-ecstore --lib disk::fs::tests::test_remove -- --nocapture"
- name: Test Windows path conversion
shell: pwsh
run: "cargo test -p rustfs-utils --no-default-features --features path --lib dunce:: -- --nocapture"
+1 -1
View File
@@ -155,7 +155,7 @@ default build (lifecycle:
- ⚠️ PARTIALLY VIOLATED: serving-side `s3s` references remain in ecstore
(bucket metadata/replication/lifecycle DTOs and error mapping). The
count is ratcheted shrink-only by `scripts/check_s3s_footprint.sh`
(`S3S_ECSTORE_FILES_BASELINE`; the `object_lock` module was converted to
(the `ecstore_files` line of `.config/s3s-footprint-baseline.txt`; the `object_lock` module was converted to
storage-level types as the first ratchet step). Target state: the
baseline reaches zero and ecstore's `Cargo.toml` drops `s3s`.
-1
View File
@@ -26,7 +26,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- IAM snapshot/deletion replay, target-assigned delete-marker purges, timestamp ordering, and best-effort peer broadcast now close the control-plane gaps found by the R6 review ([#7195](https://github.com/rustfs/rustfs/pull/7195)).
- A `set-remote-target?update=true&creds=true` request on a site-replication peer's bucket target is now rejected with `400 InvalidRequest`, naming the target and why it counts as peer-owned. It used to return success while keeping the stored endpoint, credentials, and TLS settings, so an endpoint edit from the Console's bucket replication form appeared to apply but did not. Change a peer's address with a site-replication edit instead; see "Changing a site's address" in `docs/operations/site-replication-operations.md`. The `sync`, `bandwidth`, `proxy`, and `path` update groups are unchanged.
- Site-replication edits that omit the site name, such as `mc admin replicate update --deployment-id <ID> --endpoint <URL>`, now keep the stored name on the editing site and on every receiver instead of renaming the site after the new endpoint host. No persisted format changes.
- The persisted replication MRF journal is now replayed every 5 minutes while the server runs, not only at startup. Delete-marker purges whose targets were unreachable, and replications that overflowed the worker queues, now converge within one replay interval after the target returns instead of waiting for a restart. A target that stays down is re-attempted once per retained entry per interval across the cluster: the start time of the last pass is shared through a small `config/replication/mrf.bin.replay` object, so nodes do not each replay once per interval. Older versions ignore that object; existing persisted formats are unchanged.
- Upgrade and rollback: upgrade every node in one site consecutively and verify reconciliation before moving to the next site; do not intentionally run a site mixed-version. Target-version ledger keys are harmless on rollback, although old code cannot use their routing. Before rolling back past [#7307](https://github.com/rustfs/rustfs/pull/7307), drain or repair every pending version purge: older code can free a retained version's data directory before its remote purge is acknowledged. See `docs/operations/site-replication-operations.md`.
### Security
Generated
+532 -546
View File
File diff suppressed because it is too large Load Diff
+95 -76
View File
@@ -26,6 +26,7 @@ members = [
"crates/ecstore", # Erasure coding storage implementation
"crates/e2e_test", # End-to-end test suite
"crates/filemeta", # File metadata management
"crates/gateway-adapter", # RustFS implementations of the gateway extension traits
"crates/heal", # Erasure set and object healing
"crates/heal-contracts", # Heal request/response channel contracts
"crates/iam", # Identity and Access Management
@@ -48,7 +49,7 @@ members = [
"crates/replication", # Replication contracts and wire formats
"crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
"crates/s3-client", # S3 client for engine-side consumption of remote S3 endpoints (tiering, transition targets)
"crates/s3-types", # S3 event type definitions
"crates/s3-types", # S3 event and error type definitions
"crates/s3-ops", # S3 operation definitions and mapping
"crates/s3select-api", # S3 Select API interface
"crates/s3select-query", # S3 Select query engine
@@ -75,7 +76,7 @@ edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.99.0"
version = "1.0.5"
version = "1.0.2"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -92,64 +93,76 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.5" }
rustfs-heal = { path = "crates/heal", version = "1.0.5" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.5" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.5" }
rustfs-audit = { path = "crates/audit", version = "1.0.5" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.5" }
rustfs-common = { path = "crates/common", version = "1.0.5" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.5" }
rustfs-config = { path = "./crates/config", version = "1.0.5" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.5" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.5" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.5" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.5" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.5" }
rustfs-iam = { path = "crates/iam", version = "1.0.5" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.5" }
rustfs-license = { path = "crates/license", version = "1.0.5" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.5" }
rustfs-kms = { path = "crates/kms", version = "1.0.5" }
rustfs-lock = { path = "crates/lock", version = "1.0.5" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.5" }
rustfs-notify = { path = "crates/notify", version = "1.0.5" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.5" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.5" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.5" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.5", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.5" }
rustfs-obs = { path = "crates/obs", version = "1.0.5" }
rustfs-policy = { path = "crates/policy", version = "1.0.5" }
rustfs-protos = { path = "crates/protos", version = "1.0.5" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.5" }
rustfs-replication = { path = "crates/replication", version = "1.0.5" }
rustfs-rio = { path = "crates/rio", version = "1.0.5" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.5" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.5" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.5" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.5" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.5" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.5" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.5" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.5" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.5" }
rustfs-signer = { path = "crates/signer", version = "1.0.5" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.5" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.5" }
rustfs-targets = { path = "crates/targets", version = "1.0.5" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.5" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.5" }
rustfs-utils = { path = "crates/utils", version = "1.0.5" }
rustfs-zip = { path = "./crates/zip", version = "1.0.5" }
rustfs = { path = "./rustfs", version = "1.0.2" }
rustfs-heal = { path = "crates/heal", version = "1.0.2" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.2" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.2" }
rustfs-audit = { path = "crates/audit", version = "1.0.2" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.2" }
rustfs-common = { path = "crates/common", version = "1.0.2" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.2" }
rustfs-config = { path = "./crates/config", version = "1.0.2" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.2" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.2" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.2" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.2" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.2" }
rustfs-gateway-adapter = { path = "crates/gateway-adapter", version = "1.0.2" }
rustfs-iam = { path = "crates/iam", version = "1.0.2" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.2" }
rustfs-license = { path = "crates/license", version = "1.0.2" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.2" }
rustfs-kms = { path = "crates/kms", version = "1.0.2" }
rustfs-lock = { path = "crates/lock", version = "1.0.2" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.2" }
rustfs-notify = { path = "crates/notify", version = "1.0.2" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.2" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.2" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.2" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.2", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.2" }
rustfs-obs = { path = "crates/obs", version = "1.0.2" }
rustfs-policy = { path = "crates/policy", version = "1.0.2" }
rustfs-protos = { path = "crates/protos", version = "1.0.2" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.2" }
rustfs-replication = { path = "crates/replication", version = "1.0.2" }
rustfs-rio = { path = "crates/rio", version = "1.0.2" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.2" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.2" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.2" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.2" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.2" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.2" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.2" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.2" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.2" }
rustfs-signer = { path = "crates/signer", version = "1.0.2" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.2" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.2" }
rustfs-targets = { path = "crates/targets", version = "1.0.2" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.2" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
rustfs-utils = { path = "crates/utils", version = "1.0.2" }
rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
# The gateway's S3 types: the DTOs and persisted bucket-configuration codecs the
# engine crates move onto from s3s (rustfs/backlog#2745). The one pin of the
# gateway in this workspace; no crate may enable a `compat-s3s*` feature except
# the s3s-facing edge of the rustfs binary.
rustfs-gateway-types = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.32.0", default-features = false }
rustfs-gateway-stream = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.2.0", default-features = false }
# The gateway framework itself, consumed for its extension traits by the ring-2
# adapter. `default-features = false` keeps the `server` feature, and with it the
# server crate, out of the build.
rustfs-gateway = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.57.0", default-features = false }
# Async Runtime and Networking
async-channel = "2.5.0"
async_zip = { default-features = false, version = "0.0.19" }
mysql_async = { default-features = false, version = "0.37.1" }
async-compression = { version = "0.4.52" }
async-compression = { version = "0.4.51" }
async-trait = "0.1.92"
async-nats = { version = "0.52.0", default-features = false }
async-nats = { version = "0.51.0", default-features = false }
axum = "0.8.9"
futures = "0.3.34"
futures-core = "0.3.34"
@@ -168,16 +181,16 @@ http-body-util = "0.1.5"
memchr = "2.8.3"
minlz = "1.2.3"
reqwest = "0.13.5"
rustfs-kafka-async = { version = "1.4.1" }
rustfs-kafka-async = { version = "1.4.0" }
socket2 = { version = "0.6.5" }
tokio = { version = "1.54.1" }
tokio-rustls = { default-features = false, version = "0.26.6" }
tokio-stream = { version = "0.1.19" }
tokio-test = "0.4.6"
tokio-util = { version = "0.7.20" }
tonic = { version = "0.14.9" }
tonic-prost = { version = "0.14.9" }
tonic-prost-build = { version = "0.14.9" }
tonic = { version = "0.14.8" }
tonic-prost = { version = "0.14.8" }
tonic-prost-build = { version = "0.14.8" }
tower = { version = "0.5.3" }
tower-http = { version = "0.7.1" }
h3 = "0.0.8"
@@ -191,7 +204,6 @@ quinn = { version = "0.11.12", default-features = false, features = [
# Serialization and Data Formats
apache-avro = { version = "0.22.0", features = ["snappy", "zstandard"] }
bytes = { version = "1.12.1" }
csv-core = "0.1.13"
bytesize = "2.7.0"
byteorder = "1.5.0"
flatbuffers = "25.12.19"
@@ -213,13 +225,12 @@ serde_urlencoded = "0.7.1"
# releases.
aes-gcm = { version = "0.11.1" }
argon2 = { version = "0.6.0" }
aws-lc-rs = { version = "1.19.0", default-features = false }
blake2 = "0.11.0"
chacha20poly1305 = { version = "0.11.0" }
crc-fast = "1.10.0"
hmac = { version = "0.13.0" }
jsonwebtoken = { version = "11.1.0" }
openidconnect = { default-features = false, version = "4.1.0" }
openidconnect = { default-features = false, version = "4.0" }
pbkdf2 = "0.13.0"
p256 = { version = "0.14.0", features = ["ecdsa", "pkcs8"] }
rsa = { version = "=0.10.0-rc.18" }
@@ -236,7 +247,7 @@ proptest = "1"
# Time and Date
chrono = { version = "0.4.45" }
humantime = "2.4.0"
jiff = { version = "0.2.39" }
jiff = { version = "0.2.38" }
time = { version = "0.3.55" }
# Database
@@ -249,15 +260,15 @@ anyhow = "1.0.104"
arc-swap = "1.9.3"
rustfs-tokio-tar = { version = "0.7.1" }
# Candidate Snowball parser versions exercised by rustfs-zip compatibility fixtures.
tar-codec = "0.0.16"
tar-framing = "0.0.16"
tar-codec = "0.0.15"
tar-framing = "0.0.15"
atoi = "3.1.1"
atomic_enum = "0.3.0"
aws-config = { version = "1.13.0" }
aws-credential-types = { version = "1.3.1" }
aws-sdk-kms = { default-features = false, version = "1.129.0" }
aws-sdk-s3 = { default-features = false, version = "1.157.0" }
aws-sdk-sts = { default-features = false, version = "1.124.0" }
aws-sdk-kms = { default-features = false, version = "1.128.0" }
aws-sdk-s3 = { default-features = false, version = "1.156.0" }
aws-sdk-sts = { default-features = false, version = "1.123.0" }
aws-smithy-async = { version = "1.3.1" }
aws-smithy-http-client = { default-features = false, version = "1.5.2" }
aws-smithy-runtime-api = { version = "1.19.2" }
@@ -296,7 +307,6 @@ local-ip-address = "0.6.13"
log = "0.4"
memmap2 = "0.9.12"
lz4 = "1.28.1"
lz4_flex = { version = "=0.14.0", default-features = false, features = ["std", "frame"] }
matchit = "0.9.2"
md-5 = "0.11.0"
mime_guess = "2.0.5"
@@ -322,13 +332,17 @@ rustfs-erasure-codec = { version = "9.0.0" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.34.0" }
redis = { version = "1.8.1" }
redis = { version = "1.8.0" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.6" }
rust-embed = { version = "8.13.0" }
rustc-hash = { version = "2.1.3" }
s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
# The same s3s pin under the name the legacy edge of the gateway migration uses
# (crates/s3-types, feature `compat-s3s`). Keep `rev` identical to `s3s` above.
# DELETE BY T4.2 (rustfs/backlog#2784).
legacy-s3s = { package = "s3s", git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.4"
@@ -337,10 +351,11 @@ compact_str = "0.10.0"
snap = "1.1.2"
starshard = { version = "2.4.0" }
strum = { version = "0.28.0" }
sysinfo = "0.40.1"
sync_wrapper = "1.0.2"
sysinfo = "0.40.0"
temp-env = "0.3.6"
tempfile = "3.27.0"
test-case = "3.4.1"
test-case = "3.4.0"
thiserror = "2.0.21"
tracing = { version = "0.1.44" }
tracing-appender = "0.2.5"
@@ -359,13 +374,13 @@ winapi-util = "0.1.11"
windows = { version = "0.62.2" }
windows-sys = "0.61.2"
xxhash-rust = { version = "0.8.19" }
zip = "9.3.0"
zip = "9.1.3"
zstd = "0.14.3"
# Observability and Metrics
metrics = "0.24.6"
metrics-util = "0.20"
dial9 = { version = "0.5.7", default-features = false }
dial9 = { version = "0.5.6", default-features = false }
opentelemetry = { version = "0.33.1" }
opentelemetry-appender-tracing = { version = "0.33.1" }
opentelemetry-otlp = { version = "0.33.1" }
@@ -380,16 +395,16 @@ libunftp = { version = "0.24.0" }
unftp-core = "0.1.0"
suppaftp = { version = "12.2.1" }
rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.65.1" }
russh = { version = "0.65.0" }
russh-sftp = "3.0.1"
# WebDAV
dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.7.3" }
rustfs-mimalloc = { version = "0.7.2" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.31.1", default-features = false }
hotpath = { version = "0.30.1", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.49" }
@@ -430,3 +445,7 @@ strip = "none"
# Pin upstream fixes until compatible crates.io releases are available.
[patch.crates-io]
zstd = { path = "crates/zstd-compat" }
# Preserve RustFS OIDC flows while migrating the old elliptic-curve dependency chain.
# Upstream: openidconnect-rs#244 and oauth2-rs#352.
openidconnect = { git = "https://github.com/ramosbugs/openidconnect-rs", rev = "2f77ee73bc4554e1cb6279adb386005429fb4e41" }
oauth2 = { git = "https://github.com/ramosbugs/oauth2-rs", rev = "89585f4a280afd7b285eaf1c9be0068874e03d22" }
+5 -22
View File
@@ -12,13 +12,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
ARG BINARY_SOURCE=download
FROM alpine:3.24.2 AS download
FROM alpine:3.24.2 AS build
ARG TARGETARCH
ARG RELEASE=latest
ARG RC_RELEASE=latest
RUN apk add --no-cache ca-certificates curl jq unzip
WORKDIR /build
@@ -72,17 +69,6 @@ RUN set -eux; \
chmod +x /build/rustfs; \
rm -rf rustfs.zip /build/.tmp || true
COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM alpine:3.24.2
@@ -109,17 +95,14 @@ RUN apk upgrade --no-cache && \
ca-certificates \
coreutils \
curl \
jq \
tzdata \
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /build/rustfs /usr/bin/rustfs
COPY --from=build /build/rc /usr/bin/rc
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
COPY bootstrap/ /etc/rustfs/bootstrap/
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /usr/bin/rustfs /usr/bin/rc /etc/rustfs/entrypoint.sh && \
ln -s /etc/rustfs/entrypoint.sh /entrypoint.sh
RUN chmod +x /usr/bin/rustfs /entrypoint.sh
RUN addgroup -g 10001 -S rustfs && \
adduser -u 10001 -G rustfs -S rustfs -D && \
@@ -141,6 +124,6 @@ VOLUME ["/data"]
USER rustfs
ENTRYPOINT ["/etc/rustfs/entrypoint.sh"]
ENTRYPOINT ["/entrypoint.sh"]
CMD ["rustfs"]
+4 -13
View File
@@ -12,10 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
FROM rust:1.99.1-trixie
ARG TARGETARCH
ARG RC_RELEASE=latest
FROM rust:1.97.1-trixie
RUN set -eux; \
export DEBIAN_FRONTEND=noninteractive; \
@@ -24,7 +21,6 @@ RUN set -eux; \
build-essential \
ca-certificates \
curl \
jq \
git \
pkg-config \
libssl-dev \
@@ -40,7 +36,6 @@ COPY . .
RUN ./scripts/static.sh
RUN cargo run --bin gproto
RUN cargo build --release --locked --bin rustfs
RUN sh /usr/src/rustfs/scripts/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /usr/local/bin/rc
RUN set -eux; \
groupadd -g 10001 rustfs; \
@@ -55,12 +50,8 @@ RUN set -eux; \
chown -R rustfs:rustfs /data /logs /app; \
chmod 0750 /data /logs
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
COPY bootstrap/ /etc/rustfs/bootstrap/
RUN install -m 0755 /usr/src/rustfs/target/release/rustfs /usr/bin/rustfs && \
install -m 0755 /usr/local/bin/rc /usr/bin/rc && \
chmod +x /etc/rustfs/entrypoint.sh && \
ln -s /etc/rustfs/entrypoint.sh /entrypoint.sh
COPY entrypoint.sh /entrypoint.sh
RUN install -m 0755 /usr/src/rustfs/target/release/rustfs /usr/bin/rustfs && chmod +x /entrypoint.sh
ENV RUSTFS_VOLUMES="/data" \
RUST_LOG="warn" \
@@ -72,5 +63,5 @@ ENV RUSTFS_VOLUMES="/data" \
EXPOSE 9000 9001
ENTRYPOINT ["/etc/rustfs/entrypoint.sh"]
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/usr/bin/rustfs"]
+4 -22
View File
@@ -12,13 +12,10 @@
# See the License for the specific language governing permissions and
# limitations under the License.
ARG BINARY_SOURCE=download
FROM ubuntu:26.04 AS download
FROM ubuntu:26.04 AS build
ARG TARGETARCH
ARG RELEASE=latest
ARG RC_RELEASE=latest
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
@@ -79,17 +76,6 @@ RUN set -eux; \
chmod +x /build/rustfs; \
rm -rf rustfs.zip /build/.tmp || true
COPY scripts/download_rc.sh /usr/local/bin/download_rc.sh
RUN sh /usr/local/bin/download_rc.sh "$TARGETARCH" gnu "$RC_RELEASE" /build/rc
# Candidate packaging supplies the accepted executables from verified archives.
# The default download stage remains available for local image builds.
FROM scratch AS supplied
ARG TARGETARCH
COPY .release-inputs/${TARGETARCH}/ /build/
FROM ${BINARY_SOURCE} AS build
FROM ubuntu:26.04
ARG RELEASE=latest
@@ -113,18 +99,14 @@ RUN apt-get update && apt-get upgrade -y \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
ca-certificates \
curl \
jq \
tzdata \
&& test "$(TZ=Asia/Kolkata date +%z)" = "+0530" \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /build/rustfs /usr/bin/rustfs
COPY --from=build /build/rc /usr/bin/rc
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
COPY bootstrap/ /etc/rustfs/bootstrap/
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /usr/bin/rustfs /usr/bin/rc /etc/rustfs/entrypoint.sh && \
ln -s /etc/rustfs/entrypoint.sh /entrypoint.sh
RUN chmod +x /usr/bin/rustfs /entrypoint.sh
RUN groupadd -g 10001 rustfs && \
useradd -u 10001 -g rustfs -m -s /sbin/nologin rustfs && \
@@ -149,6 +131,6 @@ VOLUME ["/data"]
USER rustfs
ENTRYPOINT ["/etc/rustfs/entrypoint.sh"]
ENTRYPOINT ["/entrypoint.sh"]
CMD ["rustfs"]
+7 -19
View File
@@ -28,19 +28,17 @@ ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETARCH
ARG RUSTFS_BUILD_FEATURES=""
ARG RC_RELEASE=latest
# -----------------------------
# Build stage
# -----------------------------
FROM rust:1.99.1-trixie AS builder
FROM rust:1.97.1-trixie AS builder
# Re-declare args after FROM
ARG TARGETPLATFORM
ARG BUILDPLATFORM
ARG TARGETARCH
ARG RUSTFS_BUILD_FEATURES
ARG RC_RELEASE
# Debug: print platforms
RUN echo "Build info -> BUILDPLATFORM=${BUILDPLATFORM}, TARGETPLATFORM=${TARGETPLATFORM}"
@@ -54,7 +52,6 @@ RUN set -eux; \
build-essential \
ca-certificates \
curl \
jq \
git \
pkg-config \
libssl-dev \
@@ -169,8 +166,6 @@ RUN --mount=type=cache,target=/usr/local/cargo/registry \
;; \
esac
RUN sh /usr/src/rustfs/scripts/download_rc.sh "$TARGETARCH" musl "$RC_RELEASE" /usr/local/bin/rc
# -----------------------------
# Development stage (keeps toolchain)
# -----------------------------
@@ -204,13 +199,10 @@ ENV RUSTFS_VOLUMES="/data" \
# We rely on cargo run to build and run
EXPOSE 9000 9001
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
COPY bootstrap/ /etc/rustfs/bootstrap/
RUN install -m 0755 /usr/local/bin/rc /usr/bin/rc && \
chmod +x /etc/rustfs/entrypoint.sh && \
ln -s /etc/rustfs/entrypoint.sh /entrypoint.sh
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/etc/rustfs/entrypoint.sh"]
ENTRYPOINT ["/entrypoint.sh"]
CMD ["cargo", "run", "--bin", "rustfs", "--"]
# -----------------------------
@@ -235,7 +227,6 @@ RUN set -eux; \
apt-get install -y --no-install-recommends \
ca-certificates \
curl \
jq \
tzdata \
coreutils; \
rm -rf /var/lib/apt/lists/*
@@ -257,11 +248,8 @@ RUN set -eux; \
# Copy the freshly built binary and the entrypoint
COPY --from=builder /usr/local/bin/rustfs /usr/bin/rustfs
COPY --from=builder /usr/local/bin/rc /usr/bin/rc
COPY entrypoint.sh /etc/rustfs/entrypoint.sh
COPY bootstrap/ /etc/rustfs/bootstrap/
RUN chmod +x /usr/bin/rustfs /usr/bin/rc /etc/rustfs/entrypoint.sh && \
ln -s /etc/rustfs/entrypoint.sh /entrypoint.sh
COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /usr/bin/rustfs /entrypoint.sh
# Default environment (override in docker run/compose as needed)
ENV RUSTFS_VOLUMES="/data" \
@@ -275,5 +263,5 @@ EXPOSE 9000
VOLUME ["/data"]
# Keep root here; entrypoint will drop privileges using chroot --userspec
ENTRYPOINT ["/etc/rustfs/entrypoint.sh"]
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/usr/bin/rustfs"]
+3 -3
View File
@@ -145,7 +145,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# Using specific version
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
```
If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -224,7 +224,7 @@ For developers who want to build RustFS Docker images from source with multi-arc
./docker-buildx.sh --push
# Build specific version
./docker-buildx.sh --release 1.0.5 --push
./docker-buildx.sh --release 1.0.4 --push
# Build for custom registry
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -242,7 +242,7 @@ You can also use Make targets for convenience:
```bash
make docker-buildx # Build locally
make docker-buildx-push # Build and push
make docker-buildx-version VERSION=1.0.5 # Build specific version
make docker-buildx-version VERSION=1.0.4 # Build specific version
make help-docker # Show all Docker-related commands
```
+3 -3
View File
@@ -138,7 +138,7 @@ chown -R 10001:10001 data logs
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
# 使用指定版本运行
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.5
docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.4
```
如果您通过绑定挂载启用 TLS 证书目录,也请用同样方式准备该目录:
@@ -176,7 +176,7 @@ docker compose -f docker-compose-simple.yml up -d
./docker-buildx.sh --push
# 构建指定版本
./docker-buildx.sh --release 1.0.5 --push
./docker-buildx.sh --release 1.0.4 --push
# 构建并推送到自定义仓库
./docker-buildx.sh --registry your-registry.com --namespace yourname --push
@@ -193,7 +193,7 @@ docker compose -f docker-compose-simple.yml up -d
```bash
make docker-buildx # 本地构建
make docker-buildx-push # 构建并推送
make docker-buildx-version VERSION=1.0.5 # 构建指定版本
make docker-buildx-version VERSION=1.0.4 # 构建指定版本
make help-docker # 显示所有 Docker 相关命令
```
-21
View File
@@ -1,21 +0,0 @@
# RustFS image bootstrap
The official image bundles this directory at `/etc/rustfs/bootstrap`. The default `buckets.json` is an empty JSON array, so the image starts RustFS directly and does not require a bootstrap endpoint or bootstrap credentials.
To create buckets, replace `buckets.json` in a derived image with a JSON array of bucket names. The entrypoint creates each bucket if it does not exist and applies `public-access-block.json`, whose default blocks all four public access paths. Edit that file in the derived image only when public access is an explicit requirement.
Optional per-bucket files use the bucket name as the filename without an extension:
| Directory | File format | Applied operation |
| --- | --- | --- |
| `policies/` | S3 bucket policy JSON | `rc bucket anonymous set-json` |
| `versioning/` | JSON boolean (`true` or `false`) | Enable or suspend bucket versioning |
| `lifecycle/` | `rc bucket lifecycle rule import` JSON | Replace lifecycle rules |
| `cors/` | JSON or XML accepted by `rc bucket cors set` | Replace CORS rules |
| `notifications/` | JSON array of `{ "arn": "...", "events": ["..."] }` | Add or replace event rules by ARN (prefix/suffix filters are not currently accepted) |
Every per-bucket file must correspond to a bucket listed in `buckets.json`. Applying these files is repeatable: bucket creation ignores existing buckets, while each configuration operation replaces or reconciles that bucket's requested state.
When `buckets.json` is non-empty, set `RUSTFS_BOOTSTRAP_ENDPOINT` to the RustFS S3 endpoint reachable from the container, for example `http://rustfs:9000` for a Compose service or the internal service DNS name in Kubernetes. The entrypoint waits on that endpoint's `/health/ready`; it never assumes `localhost`. `RUSTFS_BOOTSTRAP_TIMEOUT` controls the readiness wait in seconds (default `120`, maximum `3600`). `RUSTFS_BOOTSTRAP_DIR` can point to a different in-image configuration directory; it defaults to `/etc/rustfs/bootstrap`. Set `RUSTFS_ACCESS_KEY` and `RUSTFS_SECRET_KEY`, or their `_FILE` variants, for bootstrap authentication. The endpoint and credentials are read at runtime and are not baked into the image.
RustFS built-in IAM policies remain unchanged; the default bootstrap installs no custom bucket policies, lifecycle rules, versioning, CORS rules, or notification rules. The bundled RustFS CLI is resolved from a published release and its release checksum is verified. Set the Docker build argument `RC_RELEASE` to a version to pin it. Build the RustFS image after a CLI release containing the `bucket public-access-block` commands is available. For Kubernetes, the configured service must route traffic to the starting pod before its readiness probe succeeds (for example, with `publishNotReadyAddresses: true`); otherwise the service endpoint cannot reach the pod while the entrypoint waits for readiness.
-1
View File
@@ -1 +0,0 @@
[]
-1
View File
@@ -1 +0,0 @@
-1
View File
@@ -1 +0,0 @@
-1
View File
@@ -1 +0,0 @@
-1
View File
@@ -1 +0,0 @@
-6
View File
@@ -1,6 +0,0 @@
{
"block_public_acls": true,
"ignore_public_acls": true,
"block_public_policy": true,
"restrict_public_buckets": true
}
-1
View File
@@ -1 +0,0 @@
-100
View File
@@ -902,104 +902,4 @@ mod tests {
assert_eq!(raw.len(), 8);
assert_eq!(&raw[..], reference.digest().to_be_bytes().as_slice());
}
#[test]
fn checksums_streaming_chunks_and_header_encoding_match_oneshot_across_unaligned_boundaries() {
use crate::http::HttpChecksum;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
for &payload_len in &[0usize, 1, 31, 33, 1024, 4096 + 37, 64 * 1024 + 19] {
let payload: Vec<u8> = (0..payload_len).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_raw = one_shot.finalize();
assert_eq!(expected_raw.len(), algorithm.raw_len());
let mut one_shot_hdr = algorithm.into_impl();
one_shot_hdr.update(&payload);
let expected_headers = one_shot_hdr.headers();
for &chunk_size in &[1usize, 31, 1021, 4096] {
if payload_len > 4096 && chunk_size == 1 {
continue;
}
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(chunk);
}
let actual_headers = streaming.headers();
assert_eq!(
actual_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming mismatch at payload_len={payload_len}, chunk_size={chunk_size}"
);
}
}
}
}
#[test]
#[ignore = "utility throughput microbenchmark; run via cargo test -p rustfs-checksums --lib --release -- --ignored --nocapture"]
fn microbench_checksums_streaming_chunks_and_header_encoding() {
use crate::http::HttpChecksum;
use std::hint::black_box;
use std::time::Instant;
let all = [
ChecksumAlgorithm::Crc32,
ChecksumAlgorithm::Crc32c,
ChecksumAlgorithm::Crc64Nvme,
ChecksumAlgorithm::Sha1,
ChecksumAlgorithm::Sha256,
ChecksumAlgorithm::Sha512,
ChecksumAlgorithm::Xxhash3,
ChecksumAlgorithm::Xxhash64,
ChecksumAlgorithm::Xxhash128,
];
let payload: Vec<u8> = (0..(256 * 1024)).map(|i| ((i * 19 + 5) % 253) as u8).collect();
for &chunk_size in &[4 * 1024usize, 64 * 1024, 256 * 1024] {
for algorithm in all {
let mut one_shot = algorithm.into_impl();
one_shot.update(&payload);
let expected_headers = one_shot.headers();
let iters = 8usize;
let t0 = Instant::now();
let mut last_headers = http::HeaderMap::new();
for _ in 0..iters {
let mut streaming: Box<dyn HttpChecksum> = algorithm.into_impl();
for chunk in payload.chunks(chunk_size) {
streaming.update(black_box(chunk));
}
last_headers = streaming.headers();
black_box(&last_headers);
}
let elapsed = t0.elapsed();
assert_eq!(
last_headers.get(algorithm.http_header_name()),
expected_headers.get(algorithm.http_header_name()),
"{algorithm:?} streaming chunk_size={chunk_size} header mismatch"
);
let total_mib = (payload.len() * iters) as f64 / (1024.0 * 1024.0);
let mibs = total_mib / elapsed.as_secs_f64().max(1e-9);
println!(
"[microbench][checksums] algo={algorithm:?} chunk_size={chunk_size}B throughput={mibs:.1} MiB/s (size={}B)",
algorithm.raw_len()
);
}
}
}
}
+14 -83
View File
@@ -10,10 +10,6 @@ use std::time::Duration;
use tokio::sync::{Notify, Semaphore};
use tokio::time::Instant;
pub mod permit_set;
const MAX_PERMIT_REFERENCES: usize = usize::MAX / 2;
/// A resource owner publishes a capacity; running units retain their permits
/// after a reduction. Lease slots are reserved once, with no per-unit Arc
/// allocation. Tokio wait queues may lazily initialize platform mutex storage
@@ -40,7 +36,6 @@ pub struct AdaptiveAdmission {
maximum: u32,
slots: Box<[PermitSlots]>,
next_chunk: AtomicUsize,
completed: AtomicU64,
}
struct PermitSlots {
@@ -66,12 +61,10 @@ pub struct AdaptivePermit {
impl Clone for AdaptivePermit {
fn clone(&self) -> Self {
// Subset carriers can retain slot references through a shared owner,
// so the slot count needs its own bound in addition to Arc's guard.
// Every slot reference owns a strong owner reference. Clone the Arc
// first, so its overflow protection also bounds the slot refcount.
let owner = Arc::clone(&self.owner);
if owner.slots[self.slot / 64].references[self.slot % 64].fetch_add(1, Ordering::Relaxed) >= MAX_PERMIT_REFERENCES {
std::process::abort();
}
owner.slots[self.slot / 64].references[self.slot % 64].fetch_add(1, Ordering::Relaxed);
Self { owner, slot: self.slot }
}
}
@@ -121,22 +114,11 @@ impl NodeAdmission {
/// Publish feedback with this owner's actual completed-work counter.
pub fn update(&self, controller: &mut FeedbackController, observation: Option<FeedbackObservation>) -> FeedbackDecision {
self.update_with_probe(controller, observation, true)
}
/// Coordinate growth with other resource controllers in the same window.
/// Protective reductions never depend on permission to probe.
pub fn update_with_probe(
&self,
controller: &mut FeedbackController,
observation: Option<FeedbackObservation>,
allow_probe: bool,
) -> FeedbackDecision {
let observation = observation.map(|mut observation| {
observation.completed_units = self.completed();
observation
});
controller.update_with_probe(&self.capacity, observation, allow_probe)
controller.update(&self.capacity, observation)
}
/// Current leased scheduling cap and held async units.
@@ -267,31 +249,16 @@ impl FeedbackController {
/// Consume one sample and publish a three-second lease to the same gate.
/// If this function stops being called, admission expires independently.
pub fn update(&mut self, gate: &AdaptiveAdmission, observation: Option<FeedbackObservation>) -> FeedbackDecision {
self.update_with_probe(gate, observation, true)
}
/// Allow at most the sampling owner's selected controller to probe upward.
pub fn update_with_probe(
&mut self,
gate: &AdaptiveAdmission,
observation: Option<FeedbackObservation>,
allow_probe: bool,
) -> FeedbackDecision {
let observation = observation.map(|mut observation| {
observation.max_concurrency = observation.max_concurrency.min(gate.maximum);
observation
});
let decision = self.decide_with_probe(Instant::now(), observation, allow_probe);
let decision = self.decide(Instant::now(), observation);
gate.set_sampled_limit(decision.limit, Duration::from_secs(3));
decision
}
fn decide_with_probe(
&mut self,
now: Instant,
observation: Option<FeedbackObservation>,
allow_probe: bool,
) -> FeedbackDecision {
fn decide(&mut self, now: Instant, observation: Option<FeedbackObservation>) -> FeedbackDecision {
let Some(observation) = observation else {
self.previous = None;
self.stable_since = None;
@@ -336,8 +303,7 @@ impl FeedbackController {
FeedbackReason::NewBaseline
} else {
let since = *self.stable_since.get_or_insert(now);
if allow_probe
&& advanced
if advanced
&& now.saturating_duration_since(since) >= Duration::from_secs(8)
&& now.saturating_duration_since(self.last_change) >= Duration::from_secs(8)
&& self.limit < observation.max_concurrency
@@ -359,13 +325,6 @@ impl FeedbackController {
}
impl AdaptiveAdmission {
/// Successful async operations using this physical owner. Native clones
/// retain capacity but never count another completion.
pub fn completed(&self) -> Option<u64> {
let value = self.completed.load(Ordering::Acquire);
(value != u64::MAX).then_some(value)
}
/// Construct a bounded owner, with zero initial capacity allowed for unknown
/// input. The immutable maximum reserves at most 65,535 lease slots once;
/// acquiring or cloning a permit does not allocate a per-unit Arc.
@@ -392,7 +351,6 @@ impl AdaptiveAdmission {
maximum: maximum_u32,
slots,
next_chunk: AtomicUsize::new(0),
completed: AtomicU64::new(0),
})
}
@@ -508,21 +466,6 @@ impl AdaptiveAdmission {
self.slots[slot / 64].free.fetch_or(1_u64 << (slot % 64), Ordering::Release);
}
fn release_reference(&self, slot: usize) {
self.release_reference_with(slot, || {});
}
fn release_reference_with(&self, slot: usize, before_release: impl FnOnce()) {
if self.slots[slot / 64].references[slot % 64].fetch_sub(1, Ordering::AcqRel) != 1 {
return;
}
before_release();
// Publish the free slot before exposing another admission capacity.
self.release_slot(slot);
self.state.fetch_sub(1, Ordering::AcqRel);
self.changed.notify_one();
}
/// Wait for capacity. The future owns no permit until it returns; dropping
/// a pending future leaves capacity untouched. Wrap in the caller's existing
/// cancellation/deadline select before acquiring any storage commit locks.
@@ -550,7 +493,13 @@ impl AdaptiveAdmission {
impl Drop for AdaptivePermit {
fn drop(&mut self) {
self.owner.release_reference(self.slot);
if self.owner.slots[self.slot / 64].references[self.slot % 64].fetch_sub(1, Ordering::AcqRel) != 1 {
return;
}
// Publish the free slot before exposing another admission capacity.
self.owner.release_slot(self.slot);
self.owner.state.fetch_sub(1, Ordering::AcqRel);
self.owner.changed.notify_one();
}
}
@@ -912,24 +861,6 @@ mod tests {
}
}
#[tokio::test(start_paused = true)]
async fn shared_probe_permission_blocks_growth_but_never_protective_cuts() {
let gate = AdaptiveAdmission::new(NonZeroU16::new(8).expect("bound"), 0);
let mut controller = FeedbackController::new();
controller.update_with_probe(&gate, Some(healthy(0)), false);
for completed in 1..=20 {
tokio::time::advance(Duration::from_secs(1)).await;
assert_eq!(controller.update_with_probe(&gate, Some(healthy(completed)), false).limit, 1);
}
tokio::time::advance(Duration::from_secs(1)).await;
assert_eq!(controller.update_with_probe(&gate, Some(healthy(21)), true).reason, FeedbackReason::Probe);
assert_eq!(gate.snapshot().limit, 2);
let mut busy = healthy(22);
busy.pressure = ResourcePressure::Busy;
assert_eq!(controller.update_with_probe(&gate, Some(busy), false).limit, 1);
assert_eq!(controller.update_with_probe(&gate, None, false).limit, 0);
}
#[tokio::test(start_paused = true)]
async fn controller_grows_one_unit_per_cooldown_and_cuts_pressure_immediately() {
let gate = AdaptiveAdmission::new(NonZeroU16::new(8).expect("bounded test owner"), 0);
File diff suppressed because it is too large Load Diff
-12
View File
@@ -30,16 +30,6 @@ pub const COMMENT_KEY: &str = "comment";
pub const ENV_AUDIT_ENABLE: &str = "RUSTFS_AUDIT_ENABLE";
/// Global switch for enabling the notify module.
pub const ENV_NOTIFY_ENABLE: &str = "RUSTFS_NOTIFY_ENABLE";
/// Maximum audit entries per second for authorization denials of authenticated callers.
/// `0` disables denial auditing for that class of caller.
pub const ENV_AUDIT_DENIAL_RATE: &str = "RUSTFS_AUDIT_DENIAL_RATE";
/// Maximum audit entries per second for authorization denials of anonymous callers.
/// `0` disables denial auditing for that class of caller.
pub const ENV_AUDIT_DENIAL_ANONYMOUS_RATE: &str = "RUSTFS_AUDIT_DENIAL_ANONYMOUS_RATE";
/// Default denial audit rate for authenticated callers (entries per second).
pub const DEFAULT_AUDIT_DENIAL_RATE: u64 = 50;
/// Default denial audit rate for anonymous callers (entries per second).
pub const DEFAULT_AUDIT_DENIAL_ANONYMOUS_RATE: u64 = 5;
/// Default global audit switch (disabled by default).
pub const DEFAULT_AUDIT_ENABLE: bool = false;
/// Default global notify switch (disabled by default).
@@ -338,7 +328,5 @@ mod tests {
fn test_global_audit_notify_switch_constants() {
assert_eq!(ENV_AUDIT_ENABLE, "RUSTFS_AUDIT_ENABLE");
assert_eq!(ENV_NOTIFY_ENABLE, "RUSTFS_NOTIFY_ENABLE");
assert_eq!(ENV_AUDIT_DENIAL_RATE, "RUSTFS_AUDIT_DENIAL_RATE");
assert_eq!(ENV_AUDIT_DENIAL_ANONYMOUS_RATE, "RUSTFS_AUDIT_DENIAL_ANONYMOUS_RATE");
}
}
+9 -21
View File
@@ -184,27 +184,15 @@ const _: () = assert!(!DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT);
/// Capacity (distinct nonces) of the process-local internode RPC replay cache that enforces
/// one-time consumption of authenticated RPC signatures.
///
/// Nonces remain protected until their signed timestamp is outside the observed wall-clock
/// validity window, including after clock rollback. Current-time signatures live about five
/// minutes; the maximum accepted forward clock skew can extend that to ten minutes. A body-bound
/// v3 request consumes both its v3 and compatibility-v2 nonces, so size for receiver nonce intake,
/// not S3 request rate or CPU count.
///
/// Automatic admission budgets 13% of effective process memory for signed-time
/// hash tables, allocator overhead and simultaneous old/new tables during growth.
/// The reported count is an absolute UUID payload ceiling, not guaranteed throughput:
/// actual byte admission can reject earlier depending on timestamp distribution.
/// Tables grow with demand and are released together when their signed time expires.
/// CPU count, a historical minimum, and a fixed 32M ceiling do not constrain this budget.
/// If memory detection fails, the historical default remains a bounded count fallback.
///
/// Explicit values remain count ceilings with the historical minimum of 1,048,576
/// for configuration compatibility, and bypass automatic byte admission. That
/// compatibility floor does not apply to the automatically detected memory budget.
/// Zero never disables replay protection.
/// Overflow rejects the request without evicting a still-valid identity, increments
/// `rustfs_system_network_internode_replay_cache_overflow_total`, and returns resource
/// exhaustion rather than invalid credentials. Measure headroom before overriding.
/// The cache retains each nonce for the ~10-minute signature freshness envelope. Once peers use
/// replay-scoped v3 authentication, every authenticated RPC consumes one entry, so the steady
/// state holds roughly `authenticated RPC RPS x 601s` entries. This default is the minimum floor:
/// explicit operator values and resource-aware auto sizing both clamp upward to at least this
/// value. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
/// cache (replays are rejected before insertion, and an attacker cannot mint valid nonces without
/// the shared secret) — and increments
/// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow
/// counter means this capacity is undersized for the node's peak authenticated RPC rate.
pub const ENV_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: &str = "RUSTFS_INTERNODE_RPC_REPLAY_CACHE_CAPACITY";
pub const DEFAULT_INTERNODE_RPC_REPLAY_CACHE_CAPACITY: usize = 1_048_576;
+2 -3
View File
@@ -32,9 +32,8 @@ pub const ENV_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD: &str = "RUSTFS_OBJECT_MEDIUM_
/// Environment variable name for maximum concurrent disk reads for object operations.
/// - Purpose: Limit the number of concurrent disk read operations for object reads to prevent I/O saturation.
/// - Unit: request count (usize) for GET; estimated drive fanout for native LIST.
/// - Semantics: GET bodies and native LIST walks use separate per-instance budgets of this size.
/// A LIST consumes up to one budget, including its disk-producer cleanup. `0` disables both caps.
/// - Unit: request count (usize).
/// - Semantics: Throttling disk reads helps maintain overall system responsiveness under load.
/// - Example: `export RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=16`
/// - Note: This setting may interact with OS-level I/O scheduling and should be tuned based on hardware capabilities.
pub const ENV_OBJECT_MAX_CONCURRENT_DISK_READS: &str = "RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS";
+3 -3
View File
@@ -30,10 +30,10 @@ pub const ENV_TRUSTED_PROXY_VALIDATION_MODE: &str = "RUSTFS_TRUSTED_PROXY_VALIDA
/// Default validation mode is "hop_by_hop".
pub const DEFAULT_TRUSTED_PROXY_VALIDATION_MODE: &str = "hop_by_hop";
/// Select RFC 7239 "Forwarded" instead of "X-Forwarded-For" in legacy mode.
/// Environment variable to enable RFC 7239 "Forwarded" header support.
pub const ENV_TRUSTED_PROXY_ENABLE_RFC7239: &str = "RUSTFS_TRUSTED_PROXY_ENABLE_RFC7239";
/// Legacy mode uses X-Forwarded-For unless RFC 7239 is explicitly selected.
pub const DEFAULT_TRUSTED_PROXY_ENABLE_RFC7239: bool = false;
/// RFC 7239 support is enabled by default.
pub const DEFAULT_TRUSTED_PROXY_ENABLE_RFC7239: bool = true;
/// Environment variable for the maximum allowed proxy hops.
pub const ENV_TRUSTED_PROXY_MAX_HOPS: &str = "RUSTFS_TRUSTED_PROXY_MAX_HOPS";
-6
View File
@@ -146,9 +146,3 @@ pub const ENV_TARGET_STORE_COMPRESS: &str = "RUSTFS_TARGET_STORE_COMPRESS";
/// Queue-store compression is enabled by default to reduce disk footprint.
pub const DEFAULT_TARGET_STORE_COMPRESS: bool = true;
/// Maximum on-disk bytes in each target queue, including failed entries and pending writes.
pub const ENV_TARGET_STORE_MAX_BYTES: &str = "RUSTFS_TARGET_STORE_MAX_BYTES";
/// Queue writes fail at one GiB unless the operator sets another positive byte budget.
pub const DEFAULT_TARGET_STORE_MAX_BYTES: u64 = 1024 * 1024 * 1024;
+1 -6
View File
@@ -175,17 +175,12 @@ pub const ENV_HTTP_SOCKET_RECV_BUFFER_BYTES: &str = "RUSTFS_HTTP_SOCKET_RECV_BUF
pub const DEFAULT_HTTP_SOCKET_RECV_BUFFER_BYTES: usize = 0;
/// Environment variable for the S3 request-body inter-chunk read timeout
/// (seconds). Default: 300. Set to 0 to disable the inactivity timeout.
/// (seconds). Default: 300. Set to 0 to disable.
///
/// This bounds how long a `PutObject`/upload may wait for the *next* body chunk
/// while more bytes are still expected. It resets on every chunk, so it does not
/// penalize slow-but-progressing uploads — it only fires when a peer sends a
/// partial body and then goes silent *without* closing the connection (no EOF).
/// Independently of this knob, `PutObject` and `UploadPart` require at least
/// 64 KiB of body progress per 60 seconds of server wait (the minimum-progress
/// floor, logged as `upload_body_progress_stalled`); it bounds trickle uploads
/// that would otherwise pin upload admission indefinitely, and it stays in
/// force when this timeout is set to 0.
/// A reverse proxy or CDN that forwards a truncated body this way would
/// otherwise make RustFS wait forever for bytes that never arrive; with this
/// timeout the stalled read is aborted and logged with the received/expected
+1 -6
View File
@@ -91,12 +91,7 @@ pub const ENV_OBS_LOG_MATCH_MODE: &str = "RUSTFS_OBS_LOG_MATCH_MODE";
/// Default values for log cleanup
pub const DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES: u64 = 2 * 1024 * 1024 * 1024; // 2 GiB
pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No per-file trim by the cleaner
/// Size at which the active log file is rolled when
/// `RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES` is unset. Without a size roll
/// the active file is bounded only by the rotation period, and the cleaner
/// never touches the active file. An explicit `0` keeps size rolling off.
pub const DEFAULT_OBS_LOG_ACTIVE_FILE_ROLL_SIZE_BYTES: u64 = 256 * 1024 * 1024; // 256 MiB
pub const DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: u64 = 0; // No single file limit
pub const DEFAULT_OBS_LOG_COMPRESS_OLD_FILES: bool = true;
pub const DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL: u32 = 6;
pub const DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP: &str = "gzip";
+55 -157
View File
@@ -24,14 +24,6 @@ use std::{
time::{Duration, SystemTime},
};
/// Copy an existing cache-model value without including surrounding work.
#[inline]
pub fn clone_cache_value<T: ToOwned + ?Sized>(source: &T) -> T::Owned {
#[cfg(feature = "hotpath-alloc")]
let _copy_scope = hotpath::MeasurementGuardSync::new("cache_model.owned_copy", false, false);
source.to_owned()
}
/// Maximum amount a persisted `last_update` may lead the local wall clock before the
/// persisted timestamp is treated as untrustworthy.
///
@@ -350,7 +342,7 @@ impl UnknownTierStats {
self.diagnostics_dropped = checked_saturating_add(1, self.diagnostics_dropped, &mut self.counter_overflowed);
continue;
}
self.diagnostics.push(clone_cache_value(diagnostic));
self.diagnostics.push(diagnostic.clone());
}
}
if !self.diagnostics.is_empty() {
@@ -375,7 +367,7 @@ impl UnknownTierStats {
if !self.fits_add(other) {
return None;
}
let mut merged = clone_cache_value(self);
let mut merged = self.clone();
merged.merge(other);
Some(merged)
}
@@ -415,7 +407,7 @@ impl AllTierStats {
if st.is_empty() {
continue;
}
let entry = self.tiers.entry(clone_cache_value(tier)).or_default();
let entry = self.tiers.entry(tier.clone()).or_default();
*entry = entry.add(st);
}
}
@@ -445,7 +437,7 @@ impl AllTierStats {
.tiers
.keys()
.filter(|tier| tier.as_str() != "STANDARD" && tier.as_str() != "REDUCED_REDUNDANCY" && !known.contains(tier.as_str()))
.map(clone_cache_value)
.cloned()
.collect();
for tier in retired_tiers {
if let Some(stats) = self.tiers.remove(&tier) {
@@ -675,10 +667,6 @@ impl DataUsageInfo {
/// cycle. The timestamp fallback preserves ordering for legacy snapshots that
/// predate those fields.
pub fn data_usage_snapshot_is_newer(candidate: &DataUsageInfo, baseline: &DataUsageInfo) -> bool {
data_usage_identity_is_newer(candidate.snapshot_identity(), baseline.snapshot_identity())
}
fn data_usage_identity_is_newer(candidate: DataUsageSnapshotIdentity, baseline: DataUsageSnapshotIdentity) -> bool {
match (
candidate.scanner_epoch.zip(candidate.scanner_cycle),
baseline.scanner_epoch.zip(baseline.scanner_cycle),
@@ -701,18 +689,10 @@ fn data_usage_identity_is_newer(candidate: DataUsageSnapshotIdentity, baseline:
/// already advance the authoritative timestamp when deleting a bucket, so a
/// rollback delete/recreate fences the previous bucket incarnation too.
pub fn observed_data_usage_is_newer(observed: &DataUsageInfo, authoritative: &DataUsageInfo) -> bool {
observed_data_usage_is_newer_than_identity(observed, authoritative.snapshot_identity())
}
/// Return whether a nonconverged observation may supersede `authoritative`.
///
/// This applies the structural, exact-baseline, and generation checks of
/// [`observed_data_usage_is_newer`] without retaining the baseline's bucket maps.
pub fn observed_data_usage_is_newer_than_identity(observed: &DataUsageInfo, authoritative: DataUsageSnapshotIdentity) -> bool {
observed.usage_snapshot_converged == Some(false)
&& (observed.is_complete_bucket_usage_snapshot() || observed.is_valid_partial_snapshot())
&& observed.usage_snapshot_authoritative_baseline == Some(authoritative)
&& data_usage_identity_is_newer(observed.snapshot_identity(), authoritative)
&& observed.usage_snapshot_authoritative_baseline.as_ref() == Some(&authoritative.snapshot_identity())
&& data_usage_snapshot_is_newer(observed, authoritative)
}
/// Metadata describing the status of a disk-level data usage snapshot.
@@ -1257,10 +1237,10 @@ impl DataUsageEntry {
}
pub fn add_child(&mut self, hash: &DataUsageHash) {
if self.children.contains(&clone_cache_value(&hash.0)) {
if self.children.contains(&hash.key()) {
return;
}
self.children.insert(clone_cache_value(&hash.0));
self.children.insert(hash.key());
}
pub fn merge(&mut self, other: &DataUsageEntry) {
@@ -1277,7 +1257,7 @@ impl DataUsageEntry {
s_rep.replica_size += o_rep.replica_size;
s_rep.replica_count += o_rep.replica_count;
for (arn, stat) in o_rep.targets.iter() {
let st = s_rep.targets.entry(clone_cache_value(arn)).or_default();
let st = s_rep.targets.entry(arn.clone()).or_default();
st.pending_size += stat.pending_size;
st.replicated_size += stat.replicated_size;
st.failed_size += stat.failed_size;
@@ -1471,7 +1451,7 @@ pub fn prefix_usage_in_cache(
} else {
format!("{bucket}/{prefix}")
};
let entry = clone_cache_value(cache.get(&clone_cache_value(&hash_path(&root).0))?);
let entry = cache.get(&hash_path(&root).key())?.clone();
let usage = PrefixUsageSummary::from_entry(&flatten_entry(cache, &entry, 0)?);
@@ -1614,38 +1594,38 @@ pub struct DataUsageCache {
impl DataUsageCache {
pub fn replace(&mut self, path: &str, parent: &str, e: DataUsageEntry) {
let hash = hash_path(path);
self.cache.insert(clone_cache_value(&hash.0), e);
self.cache.insert(hash.key(), e);
if !parent.is_empty() {
let phash = hash_path(parent);
let p = {
let p = self.cache.entry(clone_cache_value(&phash.0)).or_default();
let p = self.cache.entry(phash.key()).or_default();
p.add_child(&hash);
clone_cache_value(p)
p.clone()
};
self.cache.insert(clone_cache_value(&phash.0), p);
self.cache.insert(phash.key(), p);
}
}
pub fn replace_hashed(&mut self, hash: &DataUsageHash, parent: &Option<DataUsageHash>, e: &DataUsageEntry) {
self.cache.insert(clone_cache_value(&hash.0), clone_cache_value(e));
self.cache.insert(hash.key(), e.clone());
if let Some(parent) = parent {
self.cache.entry(clone_cache_value(&parent.0)).or_default().add_child(hash);
self.cache.entry(parent.key()).or_default().add_child(hash);
}
}
pub fn find(&self, path: &str) -> Option<DataUsageEntry> {
self.cache.get(&clone_cache_value(&hash_path(path).0)).map(clone_cache_value)
self.cache.get(&hash_path(path).key()).cloned()
}
pub fn find_children_copy(&mut self, h: DataUsageHash) -> DataUsageHashMap {
clone_cache_value(&self.cache.entry(clone_cache_value(&h.0)).or_default().children)
self.cache.entry(h.string()).or_default().children.clone()
}
pub fn flatten(&self, root: &DataUsageEntry) -> DataUsageEntry {
let mut root = clone_cache_value(root);
for id in clone_cache_value(&root.children).iter() {
let mut root = root.clone();
for id in root.children.clone().iter() {
if let Some(e) = self.cache.get(id) {
let mut e = clone_cache_value(e);
let mut e = e.clone();
if !e.children.is_empty() {
e = self.flatten(&e);
}
@@ -1657,16 +1637,16 @@ impl DataUsageCache {
}
pub fn copy_with_children(&mut self, src: &DataUsageCache, hash: &DataUsageHash, parent: &Option<DataUsageHash>) {
if let Some(e) = src.cache.get(&clone_cache_value(&hash.0)) {
self.cache.insert(clone_cache_value(&hash.0), clone_cache_value(e));
if let Some(e) = src.cache.get(&hash.string()) {
self.cache.insert(hash.key(), e.clone());
for ch in e.children.iter() {
if *ch == clone_cache_value(&hash.0) {
if *ch == hash.key() {
return;
}
self.copy_with_children(src, &DataUsageHash(clone_cache_value(ch)), &Some(clone_cache_value(hash)));
self.copy_with_children(src, &DataUsageHash(ch.to_string()), &Some(hash.clone()));
}
if let Some(parent) = parent {
let p = self.cache.entry(clone_cache_value(&parent.0)).or_default();
let p = self.cache.entry(parent.key()).or_default();
p.add_child(hash);
}
}
@@ -1674,14 +1654,14 @@ impl DataUsageCache {
pub fn delete_recursive(&mut self, hash: &DataUsageHash) {
let mut need_remove = Vec::new();
if let Some(v) = self.cache.get(&clone_cache_value(&hash.0)) {
if let Some(v) = self.cache.get(&hash.string()) {
for child in v.children.iter() {
need_remove.push(clone_cache_value(child));
need_remove.push(child.clone());
}
}
self.cache.remove(&clone_cache_value(&hash.0));
self.cache.remove(&hash.string());
need_remove.iter().for_each(|child| {
self.delete_recursive(&DataUsageHash(clone_cache_value(child)));
self.delete_recursive(&DataUsageHash(child.to_string()));
});
}
@@ -1702,7 +1682,7 @@ impl DataUsageCache {
}
pub fn search_parent(&self, hash: &DataUsageHash) -> Option<DataUsageHash> {
let want = clone_cache_value(&hash.0);
let want = hash.key();
if let Some(last_index) = want.rfind('/')
&& let Some(v) = self.find(&want[0..last_index])
&& v.children.contains(&want)
@@ -1713,7 +1693,7 @@ impl DataUsageCache {
for (k, v) in self.cache.iter() {
if v.children.contains(&want) {
let found = DataUsageHash(clone_cache_value(k));
let found = DataUsageHash(k.clone());
return Some(found);
}
}
@@ -1721,7 +1701,7 @@ impl DataUsageCache {
}
pub fn is_compacted(&self, hash: &DataUsageHash) -> bool {
match self.cache.get(&clone_cache_value(&hash.0)) {
match self.cache.get(&hash.key()) {
Some(due) => due.compacted,
None => false,
}
@@ -1746,7 +1726,7 @@ impl DataUsageCache {
if self.cache.len() < limit {
return;
}
let top = clone_cache_value(&hash_path(&self.info.name).0);
let top = hash_path(&self.info.name).key();
let top_e = match self.find(&top) {
Some(e) => e,
None => return,
@@ -1773,7 +1753,7 @@ impl DataUsageCache {
}
pub fn reduce_children_of(&mut self, path: &DataUsageHash, limit: usize, compact_self: bool) {
let e = match self.cache.get(&clone_cache_value(&path.0)) {
let e = match self.cache.get(&path.key()) {
Some(e) => e,
None => return,
};
@@ -1783,13 +1763,13 @@ impl DataUsageCache {
}
if e.children.len() > limit && compact_self {
let mut flat = self.size_recursive(&clone_cache_value(&path.0)).unwrap_or_default();
let mut flat = self.size_recursive(&path.key()).unwrap_or_default();
flat.compacted = true;
self.delete_recursive(path);
self.replace_hashed(path, &None, &flat);
return;
}
let total = self.total_children_rec(&clone_cache_value(&path.0));
let total = self.total_children_rec(&path.key());
if total < limit {
return;
}
@@ -1802,12 +1782,12 @@ impl DataUsageCache {
let mut candidate_index = 0;
while remove > 0 && candidate_index < candidates.len() {
let e = &candidates[candidate_index];
let candidate = clone_cache_value(&e.path);
let candidate = e.path.clone();
if candidate == *path && !compact_self {
break;
}
let removing = self.total_children_rec(&clone_cache_value(&candidate.0));
let mut flat = match self.size_recursive(&clone_cache_value(&candidate.0)) {
let removing = self.total_children_rec(&candidate.key());
let mut flat = match self.size_recursive(&candidate.key()) {
Some(flat) => flat,
None => {
candidate_index += 1;
@@ -1844,7 +1824,7 @@ impl DataUsageCache {
if o.root().is_none() {
return;
}
*self = clone_cache_value(o);
*self = o.clone();
return;
};
@@ -1857,8 +1837,7 @@ impl DataUsageCache {
}
existing_root.merge(&other_root);
self.cache
.insert(clone_cache_value(&hash_path(&self.info.name).0), existing_root);
self.cache.insert(hash_path(&self.info.name).key(), existing_root);
let root_hash = self.root_hash();
for key in other_root.children.iter() {
@@ -1869,7 +1848,7 @@ impl DataUsageCache {
if let Some(existing) = self.cache.get_mut(key) {
existing.merge(&flat);
} else {
self.replace_hashed(&DataUsageHash(clone_cache_value(key)), &Some(clone_cache_value(&root_hash)), &flat);
self.replace_hashed(&DataUsageHash(key.clone()), &Some(root_hash.clone()), &flat);
}
}
}
@@ -1913,7 +1892,7 @@ impl DataUsageCache {
for (arn, stat) in rs.targets.iter() {
bui.replication_info.insert(
clone_cache_value(arn),
arn.clone(),
BucketTargetUsageInfo {
replication_pending_size: stat.pending_size,
replicated_size: stat.replicated_size,
@@ -1926,7 +1905,7 @@ impl DataUsageCache {
);
}
}
buckets_usage.insert(clone_cache_value(bucket_name), bui);
buckets_usage.insert(bucket_name.clone(), bui);
}
DataUsageInfo {
@@ -1958,20 +1937,20 @@ struct Inner {
}
fn add(data_usage_cache: &DataUsageCache, path: &DataUsageHash, candidates: &mut Vec<Inner>) -> usize {
let e = match data_usage_cache.cache.get(&clone_cache_value(&path.0)) {
let e = match data_usage_cache.cache.get(&path.key()) {
Some(e) => e,
None => return 0,
};
let mut objects = e.objects;
for ch in e.children.iter() {
objects += add(data_usage_cache, &DataUsageHash(clone_cache_value(ch)), candidates);
objects += add(data_usage_cache, &DataUsageHash(ch.clone()), candidates);
}
// Collect internal nodes (with children) as compaction candidates.
// Leaf nodes have no children to remove, so compacting them is a no-op.
if !e.children.is_empty() {
candidates.push(Inner {
objects,
path: clone_cache_value(path),
path: path.clone(),
});
}
objects
@@ -1979,7 +1958,7 @@ fn add(data_usage_cache: &DataUsageCache, path: &DataUsageHash, candidates: &mut
fn mark(duc: &DataUsageCache, entry: &DataUsageEntry, found: &mut HashSet<String>) {
for k in entry.children.iter() {
found.insert(clone_cache_value(k));
found.insert(k.to_string());
if let Some(ch) = duc.cache.get(k) {
mark(duc, ch, found);
}
@@ -2429,11 +2408,11 @@ impl SizeSummary {
}
for entry in &other.size_reconciliation {
self.record_size_reconciliation(clone_cache_value(entry));
self.record_size_reconciliation(entry.clone());
}
self.size_reconciliation_truncated |= other.size_reconciliation_truncated;
for scope in &other.reconciliation_scopes {
self.record_reconciliation_scope_with(&scope.bucket, &scope.object, || clone_cache_value(scope));
self.record_reconciliation_scope(&scope.bucket, &scope.object);
}
}
@@ -2458,13 +2437,6 @@ impl SizeSummary {
/// Mark one object scope as refreshed. Duplicate scopes are suppressed so
/// merging summaries remains bounded and deterministic.
pub fn record_reconciliation_scope(&mut self, bucket: &str, object: &str) {
self.record_reconciliation_scope_with(bucket, object, || SizeReconciliationScope {
bucket: bucket.to_string(),
object: object.to_string(),
});
}
fn record_reconciliation_scope_with(&mut self, bucket: &str, object: &str, build: impl FnOnce() -> SizeReconciliationScope) {
if !self
.reconciliation_scopes
.iter()
@@ -2474,7 +2446,10 @@ impl SizeSummary {
self.size_reconciliation_truncated = true;
return;
}
self.reconciliation_scopes.push(build());
self.reconciliation_scopes.push(SizeReconciliationScope {
bucket: bucket.to_string(),
object: object.to_string(),
});
}
}
}
@@ -3152,17 +3127,12 @@ mod tests {
};
assert!(observed_data_usage_is_newer(&observed, &authoritative));
assert!(observed_data_usage_is_newer_than_identity(&observed, authoritative.snapshot_identity()));
authoritative.last_update = Some(SystemTime::UNIX_EPOCH + Duration::from_secs(601));
assert!(
!observed_data_usage_is_newer(&observed, &authoritative),
"an old-binary namespace mutation must fence the prior bucket incarnation regardless of clock skew"
);
assert!(
!observed_data_usage_is_newer_than_identity(&observed, authoritative.snapshot_identity()),
"a captured identity must preserve the exact namespace fence"
);
}
#[test]
@@ -3189,18 +3159,6 @@ mod tests {
assert!(!observed_data_usage_is_newer(&candidate(2, 9, Some(false), true), &authoritative));
assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(true), true), &authoritative));
assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(false), false), &authoritative));
for (observed, expected) in [
(candidate(2, 11, Some(false), true), true),
(candidate(2, 9, Some(false), true), false),
(candidate(2, 11, Some(true), true), false),
(candidate(2, 11, Some(false), false), false),
] {
assert_eq!(
observed_data_usage_is_newer_than_identity(&observed, authoritative.snapshot_identity()),
expected,
"observation identity selection must preserve convergence, completeness, and generation gates"
);
}
let mut partial = candidate(2, 11, Some(false), false);
partial.usage_snapshot_partial = true;
@@ -3215,30 +3173,6 @@ mod tests {
segment_invalidation_proof: None,
}];
assert!(observed_data_usage_is_newer(&partial, &authoritative));
assert!(observed_data_usage_is_newer_than_identity(&partial, authoritative.snapshot_identity()));
}
#[test]
fn observation_identity_selection_keeps_legacy_timestamp_ordering() {
let baseline = DataUsageSnapshotIdentity {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(10)),
// An epoch without a cycle is not a coordinated position.
scanner_epoch: Some(2),
scanner_cycle: None,
};
let mut observed = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(11)),
scanner_epoch: Some(1),
usage_snapshot_complete: true,
usage_snapshot_converged: Some(false),
usage_snapshot_authoritative_baseline: Some(baseline),
..Default::default()
};
assert!(observed_data_usage_is_newer_than_identity(&observed, baseline));
observed.last_update = baseline.last_update;
assert!(!observed_data_usage_is_newer_than_identity(&observed, baseline));
observed.last_update = None;
assert!(!observed_data_usage_is_newer_than_identity(&observed, baseline));
}
#[test]
@@ -3470,42 +3404,6 @@ mod tests {
assert_eq!(summary1.versions, 15);
}
#[test]
fn test_reconciliation_scope_builds_only_after_admission() {
let mut summary = SizeSummary::new();
let builds = std::cell::Cell::new(0);
let build = || {
builds.set(builds.get() + 1);
SizeReconciliationScope {
bucket: "bucket".to_string(),
object: "object".to_string(),
}
};
summary.record_reconciliation_scope_with("bucket", "object", build);
assert_eq!(builds.get(), 1, "an admitted scope is copied once");
assert_eq!(summary.reconciliation_scopes.len(), 1);
assert_eq!(summary.reconciliation_scopes[0].bucket, "bucket");
assert_eq!(summary.reconciliation_scopes[0].object, "object");
summary.record_reconciliation_scope_with("bucket", "object", build);
assert_eq!(builds.get(), 1, "duplicate scopes must not be copied");
assert_eq!(summary.reconciliation_scopes.len(), 1);
assert!(!summary.size_reconciliation_truncated);
for index in 1..1024 {
summary.record_reconciliation_scope("bucket", &index.to_string());
}
summary.record_reconciliation_scope_with("bucket", "object", build);
assert_eq!(builds.get(), 1, "a duplicate at capacity must not be copied");
assert!(!summary.size_reconciliation_truncated);
summary.record_reconciliation_scope_with("bucket", "overflow", build);
assert_eq!(builds.get(), 1, "a rejected scope must not be copied");
assert_eq!(summary.reconciliation_scopes.len(), 1024);
assert!(summary.size_reconciliation_truncated);
}
#[test]
fn test_size_histogram_compat_rollup_sums_all_sub_buckets() {
let mut hist = SizeHistogram::default();
-1
View File
@@ -25,7 +25,6 @@ workspace = true
[features]
default = []
scanner-peer-mixed-version = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",

Some files were not shown because too many files have changed in this diff Show More