Compare commits

...

622 Commits

Author SHA1 Message Date
Chris An 1000860cb8 refactor(admin): drop the legacy target descriptor JSON response helper (#8968)
`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.
2026-10-27 00:15:37 +08:00
Chris An b783f404ad refactor(admin): migrate identity and cluster handlers to owned requests (#8966)
* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.
2026-10-26 22:23:47 +08:00
Chris An 11a991c777 refactor(admin): complete the owned admin request boundary (#8949)
* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes
2026-10-26 17:04:02 +08:00
Chris An 3f2b6f690a refactor(admin): migrate health and prefix usage to owned requests (#8887)
* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request
2026-10-26 07:34:31 +08:00
overtrue 88b626e7a0 chore(gateway): merge main into integration
Source: main 0f24511535
Target: gateway/integration 8cf6187ff4

Resolve the e2e-full selection digest conflict by recomputing both
platform digests over the merged nextest listing (685 darwin / 693
linux): main adds two rpc_fault_proxy unit tests, integration keeps its
dual-stack differential and anonymous-access additions. Raise the rustfs
test-count baseline row to the merged count (5932).
2026-10-25 09:58:02 +08:00
Hauser 0f24511535 test(e2e): harden multipart fault and TLS acceptance (#8907)
* test(e2e): tighten scoped multipart fault acceptance

Target Abort root deletion instead of child transaction cleanup, require
actual metadata peer hits, and keep faulted APIs single-attempt. Validate
bounded ordinary recovery and safe quorum refusal for partial mutations,
with the recovery enhancement tracked in backlog#2940.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(e2e): drive concurrent TLS reads independently

Give each coordinator its own supervised task and cooperative polling
budget while retaining the existing workload, deadline and byte/hash
assertions. Preserve node logs after fixture teardown and report the
operation, response phase and received bytes when a transfer times out.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 04:54:35 +08:00
Hauser 9517db6cf7 fix(listing): honor per-object quorum with borrowed metadata (#8901)
* fix(listing): honor persisted erasure quorum

Separate metadata admission and primary-reader sizing from each version's
persisted erasure requirements. Propagate agreed-entry uncertainty and
report LIST read-quorum failures as storage unavailability.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(listing): borrow metadata during quorum resolution

Keep validated metadata inspection and agreed-version selection in
filemeta. Resolve independent reader votes through borrowed slots and
retain owned buffers until a filtered result is required.

Preserve fixed-baseline legacy comparisons, cached latest fast paths,
quorum policy, and disk fan-out. Add ownership regressions and paired
release benchmarks for cold and cached version histories.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): remove redundant fallback clone

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 04:49:31 +08:00
Hauser 3bcba78ccb chore(deps): refresh allocator, profiling, and archive crates (#8910)
Run two cargo update and cargo upgrade rounds from origin/main.
Update mimalloc to 0.7.2, hotpath to 0.29.1, and ZIP to 9.1.0,
with compatible crypto, compression, and smallvec updates.
Retain the generic-array version required by crypto-common 0.1.7.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 04:49:20 +08:00
Hauser af4ce32528 perf(ecstore): reduce scanner metadata path allocation overhead (#8899)
* fix(scripts): collect scanner host telemetry on macOS

Use platform-supported iostat arguments and expose optional collector
failures without interrupting scanner-status collection.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): reuse validated metadata read path buffers

Share the checked bucket/object tuple across metadata read consumers and borrow clean normalization inputs without skipping filesystem validation. Keep Windows reconstruction and the second bucket check intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scripts): finalize scanner telemetry for failed runs

Initialize host telemetry artifacts before external captures so reused output directories cannot retain prior successes. Share normal and error-path finalization, reap collectors on request failure, and preserve the primary exit status even if status recording fails.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): distinguish unreadable recovery inventories

Treat only the exact system-prefix read-quorum result as incomplete during bounded polling. Preserve strict final reads and the shutdown/drain assertions, record failed observations, and cover interrupted empty streaks and unrelated errors.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 04:23:05 +08:00
Chris An 32c4593587 fix(heal): retain MRF state when bucket identity is unresolved (#8905)
* fix(heal): retain MRF state when bucket identity is unresolved

* test(heal): give real-storage replay fixture a bounded stack
2026-10-25 03:58:46 +08:00
Jim Chen 2f90fb19f5 fix(rio): ZigZag-encode signed delta varints and widen scratch buffer in compress_index (#8893)
* fix(rio): ZigZag-encode signed delta varints and widen scratch buffer in compress_index

Index::append_to passes signed delta residuals (c_off and u_off, which are negative whenever a compressed block is smaller than est_block_uncomp / 2) to write_varint(buf, v: i64), which checks while v >= 0x80 without ZigZag encoding and truncates every negative delta to a single byte, while tmp = [0u8; 8] panics out-of-bounds on 9-10 byte varints.

Root Cause:
In crates/rio/src/compress_index.rs, Index::append_to delta-encodes u_off and c_off (c_off -= prev.compressed_offset + c_predict, where c_predict starts at est_block_uncomp / 2) as signed i64 residuals and passes them to write_varint(buf, v: i64). However, write_varint checks while v >= 0x80 on the signed i64 without ZigZag encoding (unlike Go binary.PutVarint / binary.Varint in the S2 index format): for any negative v, v >= 0x80 is false, so write_varint writes a single truncated byte v as u8 without continuation bits, either silently corrupting compressed_offset on Index::load (when (v as u8) < 0x80) or desynchronizing the varint stream (when (v as u8) >= 0x80). Additionally, tmp = [0u8; 8] at line 223 panics with an out-of-bounds index when encoding 9-10 byte varints, and read_varint lacks an overflow check on 10-byte inputs.

Fix:
Use ZigZag encoding and decoding in write_varint and read_varint, reject 64-bit varint overflows in read_varint, and widen the scratch buffer tmp in Index::append_to from [0u8; 8] to [0u8; 10].

Signed-off-by: czcbangkai <czcbangkai@google.com>

* test(rio): use est_block_uncomp field and compare IndexInfo fields in compress_index test

---------

Signed-off-by: czcbangkai <czcbangkai@google.com>
2026-10-25 03:58:33 +08:00
Chris An 17aa75a175 fix(listing): keep the limit flag monotonic and fix metacache helpers (#8906)
`LocalDisk::walk_dir` stored its own `limit_reached` into the
`producer_limit_reached` flag that every producer of a page shares, so a
walk reaching EOF after a sibling stopped at its per-disk limit cleared the
signal and the collector reported a complete page. Use `fetch_or` so the
flag answers "did any bounded walk stop early", whatever the finish order.

`MetaCacheEntry::is_in_dir` stripped the directory prefix repeatedly, so
`a/a/b` counted as a direct child of `a/`. `MetaCacheEntriesSorted::
forward_past` kept every entry when none sorted after the marker and split
on an index computed over the flattened entries. Both are masked today by
the collector's pre-filtering; fix them and pin the contracts with tests.
2026-10-25 02:24:41 +08:00
Hauser d0f8475924 fix(multipart): recover aborts without retaining topology locks (#8904)
* fix(multipart): allow abort after partial commit

Read staging metadata at read quorum during Abort while retaining the
delete write quorum and generation checks. Bound post-fault lock recovery,
isolate quorum rows from peer breaker history, and cover the metadata and
deletion boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(multipart): release topology guards before metadata I/O

Read multipart metadata from owned disk snapshots so slow I/O does not
block registry renewal. Cover read and write lookups with a deterministic
paused-I/O regression.

Reuse one cluster per quorum matrix after observing fresh responses from
every peer. Preserve fault-hit and quorum assertions without repeatedly
starting clusters or disabling peer request breakers.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 02:01:12 +08:00
overtrue 8cf6187ff4 chore(gateway): merge main into integration 2026-10-25 01:50:03 +08:00
Hauser 7130f25af5 fix(ecstore): unblock delete replay and parallelize preparation (#8896)
* perf(ecstore): avoid head-of-line blocking in delete recovery

Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): bound durable delete preparation and bootstrap recovery

Prepare independent Linux delete transactions with bounded concurrency while retaining per-key result order and serial fallback for overlapping, rollback, and legacy requests. Preserve startup discovery until bucket metadata is initialized, and add release performance and process-crash acceptance runners.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): verify clean exits in recovery acceptance

Reject forced or OOM exits when measuring clean startup and verify original ETags across mixed-layout crash recovery and a second restart.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): distinguish prepared and decided batch crashes

Observe real preparation and receipt states separately, retain non-vacuous acknowledged-delete sentinels, and require read-only physical convergence before S3 reads can trigger cleanup.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): move final delete recovery options

Move the options value into the batch-ordering regression's final case
instead of cloning it. Keep the earlier clone needed for reuse.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): replay delete receipts after interrupted proof GC

Retire only the exact empty transaction leaf before journal activation. Preserve occupied directories and prefix children, and cover partial cross-disk proof GC without a fresh delete. Tighten crash evidence to distinguish directory-only remnants from retained content.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 01:12:34 +08:00
Hauser 900d082764 fix(scanner): make startup responsive and skip redundant config work (#8898)
* fix(scanner): bound implicit startup delay and publish config early

Limit warm-cache startup to the smaller of the cycle interval and one
minute before jitter, while preserving explicit startup settings.
Publish configuration gauges when applying runtime settings and retain
timeout diagnostics until an actual new cycle begins.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): unify startup waits and skip unchanged configuration

Bound read-only startup inspection and make startup and pre-leadership
retries respond to cancellation and configuration changes. Preserve one
startup deadline and discover replication behind lifecycle-only buckets.
Avoid irrelevant bucket listings and repeated per-disk reconfiguration.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): retain effective startup policy and fallback metrics

Keep the sampled startup deadline when configuration changes leave its
normalized policy unchanged. Initialize retained runtime metrics after
an invalid first refresh without suppressing the validation error or
replacing a newer accepted configuration.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 01:01:37 +08:00
Chris An b23723a51f feat(heal): add live node admission feedback (#8897)
* feat(heal): add live node admission feedback

* test(heal): use storage facade in admission regression

* fix(heal): preserve retryable node admission deferrals
2026-10-25 01:42:02 +09:00
Hauser 07b7d4fc18 fix(s3): preserve legacy plaintext multipart ETags in listings (#8900)
* fix(s3): preserve legacy plaintext multipart ETags in listings

Use stored object encryption markers for keyless LIST classification and rebuild persistent snapshots that discarded plaintext multipart ETags.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(s3): classify listing encryption metadata without allocations

Use fixed SSE header matching with the historical Unicode fallback. Fuse encrypted ETag proof validation into one metadata pass and clone snapshot ETags only after visibility filtering.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 01:41:03 +09:00
Jim Chen d60ef4873a fix(ecstore): avoid usize underflow when root_disk drives are offline or unknown (#8879)
get_online_offline_disks_stats() counts and demotes all root_disk entries regardless of drive state, underflowing online_disks[ep] (0usize - 1) and double-counting offline_disks[ep] when a root_disk drive is offline or unknown alongside healthy non-root drives.

Root Cause:
In crates/ecstore/src/diagnostics/admin_server_info.rs: (1) root_disk_count counts every drive with root_disk = true even if the drive was already classified into offline_disks or unknown_disks; (2) the subsequent demotion loop unconditionally decrements online_disks[ep] and increments offline_disks[ep] for every disk.root_disk without checking whether disk.state was classified as online (ok or unformatted). When an endpoint has an offline or unknown root_disk drive and zero online drives, *online_disks.get_mut(ep) -= 1 underflows 0usize (panicking in debug/checked builds or wrapping to usize::MAX) and inflates offline_disks.

Fix:
Introduce is_online_drive_state() and only count and demote root_disk drives whose state was classified as online (ok or unformatted), preserving the offline and unknown bucket tallies and keeping online + offline + unknown equal to disks_info.len().

Signed-off-by: czcbangkai <czcbangkai@google.com>
2026-10-25 00:29:15 +08:00
Hauser aaeedcd1d7 perf(rpc): remove stream boxing and verify distributed TLS reads (#8885)
* perf(rpc): remove redundant boxing from read streams

Keep the capped read stream concrete until StreamingBlob owns its type erasure. Add verified four-node TLS coverage for full, unaligned range and concurrent reads, with bounded client memory and explicit integrity checks.

Document protocol comparison commands and the old-node workaround during rolling upgrades.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): preserve TLS trust for alternate credentials

Derive credential-specific clients from the cluster configuration so they retain the generated CA. Exercise successful trusted reads and reject an invalid replacement identity instead of inheriting root credentials.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): collapse acceptance report path guard

Combine the optional report path and existing-file checks into a let chain
without changing the report or PID sidecar collision behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-25 00:12:59 +08:00
Jim Chen 8ebec9e71c fix(server): preserve UTF-8 boundaries in Content-Disposition filename parsing (#8880)
HttpCompressionConfig::extract_filename_from_content_disposition() calls header_value.to_lowercase() and slices &header_value[start..] using the byte index from lower.find("filename="), panicking on multi-byte Unicode characters whose UTF-8 byte length changes under case mapping and matching unanchored *filename= parameter suffixes.

Root Cause:
In rustfs/src/server/compress.rs, extract_filename_from_content_disposition() computes lower = header_value.to_lowercase() and uses the byte offset idx from lower.find("filename=") to slice &header_value[start..]. Because Unicode case mapping can expand or shrink UTF-8 byte lengths (e.g., U+0130 "İ" is 2 bytes in UTF-8 but lowercases to 3 bytes; U+212A "K" is 3 bytes in UTF-8 but lowercases to 1 byte), start can land inside a multi-byte UTF-8 codepoint or past the end of header_value, causing a runtime panic. In addition, lower.find("filename=") matches any parameter ending in filename= (such as x-filename=).

Fix:
Use header_value.to_ascii_lowercase() so byte offsets remain 1:1 with header_value, and require filename= to appear either at the start of the header or after a semicolon parameter delimiter.

Signed-off-by: czcbangkai <czcbangkai@google.com>
2026-10-24 23:49:24 +09:00
Jim Chen cced213b66 fix(ecstore): invalidate FdCache on in-place writes, rename_part, and directory rename_file (#8889)
StdBackend::pread_bytes caches FdCacheEntry { file, len } and skips fstat on cache hits, passing cached len directly to unsafe { mmap_options.map(file) }. However, StdBackend::open_write / UringBackend::open_write (create_file, append_file), LocalDisk::write_all_private (write_all), LocalDisk::rename_part, and directory LocalDisk::rename_file fail to invalidate FdCache, causing mmap past EOF (SIGBUS), false DiskError::FileCorrupt on grown files, or stale reads after renames.

Root Cause:
In crates/ecstore/src/disk/local.rs, StdBackend::pread_bytes caches open File handles and their initial fstat byte length in FdCache and uses the cached len directly for bounds checks and unsafe { mmap_options.map(file) }. While LocalDisk::rename_file (non-directory) and LocalDisk::delete invalidate FdCache, five mutation paths omit invalidation: (1) StdBackend::open_write (called by create_file with O_TRUNC and append_file), (2) LocalDisk::write_all_private (called by write_all with O_TRUNC), (3) LocalDisk::rename_part, (4) LocalDisk::rename_file when src_is_dir is true, and (5) UringBackend::open_write / unlink / rename which fail to forward invalidations to self.inner. Truncating a cached file across a page boundary causes mmap to map beyond EOF and trigger SIGBUS; growing a cached file causes false DiskError::FileCorrupt.

Fix:
Invalidate FdCache in StdBackend::open_write, UringBackend::open_write/unlink/rename, LocalDisk::write_all_private, LocalDisk::rename_part (both src and dst), and directory LocalDisk::rename_file (via invalidate_prefix on src and dst).

Signed-off-by: czcbangkai <czcbangkai@google.com>
2026-10-24 23:49:20 +09:00
Hauser 882217221e fix(lock): reach quorum behind silent voters (#8895)
* test(e2e): add MPU RPC quorum and recovery acceptance

Exercise separate storage and lock RPC delays across multipart phases,
including quorum boundaries, cancellation, bucket recreation, and bounded
eight-node load. Honor TMPDIR for isolated cluster drive directories.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(lock): probe healthy voters behind silent peers

Spread additional probes across the existing acquisition attempt budget
when the initial quorum-plus-spare wave stalls. Preserve quorum thresholds,
fast-path fan-out, and existing ownership and late-response cleanup.

Cover every four-node read-voter ordering, a stalled eight-node write
quorum, and normal fan-out. Extend MPU acceptance to EC12+4 at twelve and
eleven metadata acknowledgements.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-24 22:30:31 +08:00
Chris f279cefa9c feat(observability): track foreground response-body completion windows (#8894)
* feat(observability): track foreground response-body completion windows

* docs: correct foreground ownership spelling
2026-10-24 23:14:07 +09:00
Chris 8aba025bd6 feat(concurrency): retain resource leases through native I/O (#8886)
feat(concurrency): retain resource leases across owned disk work
2026-10-24 21:19:06 +08:00
Hauser 40837f67f8 perf(ecstore): avoid head-of-line blocking in delete recovery (#8884)
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-24 20:35:29 +08:00
cxymds 042f6ba3aa fix(quota): persist lazily created populated shards (#8883) 2026-10-24 21:02:52 +09:00
Chris 63eac7a387 feat(observability): sample live cgroup resource limits (#8882) 2026-10-24 20:42:20 +09:00
Chris e13336a116 feat(scanner): expose publication and heal observation status (#8878) 2026-10-24 20:04:57 +09:00
Hauser 8d5af5981d fix(s3): preserve aws-chunked error responses (#8875)
* fix(s3): preserve aws-chunked error responses

Map named aws-chunked failures through the shared bounded body classifier,
keeping typed source precedence and the dependency's S3 response contract.
Cover exact responses, upload publication, and valid signed/trailer limits.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(s3): classify upload stream errors once per cause

Reuse the typed UploadStreamError result for digest and incomplete-body
classification while preserving the shared classifier's marker priorities.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): refresh aws-chunked profile selections

Record the two new aws-chunked regression cases in smoke and full profiles.
Verify exact additions against the CI Linux inventory and native Darwin
listings, recovering each previous digest when those two cases are removed.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-24 20:04:18 +09:00
Hauser f53c5d41d4 perf(ecstore): avoid heap allocation for shard descriptors (#8873)
Keep inline shard references through both encode backends and reconstruction
verification, retaining heap spill for wider layouts and existing codec bytes.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-24 18:11:09 +08:00
Chris 4096d05ca7 test(ilm): isolate fixture recovery loop ownership (#8881) 2026-10-24 18:47:50 +09:00
Hauser d2c1434a02 fix: isolate internode reads and migrate crypto dependencies (#8872)
* fix(rio): isolate internode streams with pooled HTTP/1.1

Default the internode HTTP data plane to reusable HTTP/1.1 connections so paused shard bodies cannot consume shared HTTP/2 receive credit. Retain explicit auto negotiation and the existing TLS, pooling, and recovery paths.

Refs rustfs/rustfs#8843 and rustfs/backlog#2934.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(rpc): frame bounded shard reads for connection reuse

Preserve the bounded read cap and stream-to-EOF sentinel while making connection completion independent of a chunked terminator. Cover exact reads without EOF polling, pooled TLS reuse, and partial-body truncation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(deps): migrate signer and OIDC to patched crypto

Pin the requested signer fix and upstream OIDC/OAuth2 crypto migrations to immutable commits. Preserve registry SDK type identity, remove the old bigint and elliptic-curve chains, and verify independent login state, nonce, and PKCE values.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-24 17:41:55 +08:00
Chris c8f14dc337 test(scanner): attest running benchmark process identity (#8876)
* test(scanner): attest running benchmark process identity

* fix(scanner): reject aliased process receipt data roots

* fix(scanner): attest scheduled scan configuration
2026-10-24 18:22:11 +09:00
Chris 90c5050a0f fix(scanner): preserve unknown catch-up state on ledger failure (#8877) 2026-10-24 18:16:42 +09:00
Chris 334e8d9d3e test(scanner): verify retained versions with an independent oracle (#8870)
* test(scanner): add independent version content oracle

* test(ecstore): drain guarded delete fixture rename tails

* docs(scanner): record SDK seeding requirements
2026-10-24 18:12:06 +09:00
Ramakrishna Chilaka d8e8f6bfa4 perf(ecstore): coalesce inline PUT commit and bound disk namespace workers (#8871)
* perf(ecstore): coalesce inline PUT commit and bound disk namespace workers

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a

* fix(ecstore): drop PreparedRenameSource before post-commit rollback on Windows

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
2026-10-24 17:25:53 +09:00
Chris 4e64e86a29 feat(scanner): count object metadata visits (#8874)
* feat(scanner): count object metadata visits

* test(ecstore): drain guarded delete fixture rename tails
2026-10-24 17:24:17 +09:00
Chris a0a0cee9eb refactor(auth): derive policy conditions from AuthzFacts (#8442)
* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline
2026-10-24 16:56:01 +09:00
cozima d229c79e38 fix(s3): apply bucket CORS to virtual-hosted-style requests (#8860)
ConditionalCorsLayer took the bucket from the first path segment only.
With RUSTFS_SERVER_DOMAINS set, a virtual-hosted-style request such as
OPTIONS http://mybucket.localhost:9000/x.jpg was evaluated against a
bucket named "x.jpg", so a preflight never matched the bucket's CORS
configuration and returned 200 without Access-Control-* headers, while
the same request in path style succeeded.

Resolve the bucket from the host prefix first, reusing the virtual-host
matching that the API rate limiter already uses, and fall back to the
first path segment for path-style requests. A bucket-level request on a
virtual host ("/") is now treated as a bucket request instead of the
server root.

Co-authored-by: cozima <8464261+cozima@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-24 16:53:36 +09:00
Hauser 0c8b05585e fix(scanner): classify erasure candidates in one budgeted walk (#8869)
* test(scanner): isolate async checkpoint log capture

Scope captured tracing subscribers to each future poll and cover the
competing-dispatcher failure without weakening checkpoint assertions.
Verify exact bounded erasure-probe accounting around the 64-entry limit.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): classify erasure candidates in one budgeted walk

Continue directory classification in cooperative 64-entry batches until
part evidence or EOF, retaining normal checkpoint and namespace work.
Defer metadata accounting and namespace reuse until classification ends.
Distinguish vanished opens from incomplete iteration and preserve quota,
sampling, failed-object and non-destructive heal behavior.

Pin test callsite interest alongside poll-scoped capture to prevent an
unsubscribed sibling from caching the checkpoint callsite as disabled.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-24 15:49:00 +08:00
Hauser 911ea265c5 fix(storage): apply and index explicit delete-marker removals (#8854)
* fix(ecstore): purge every explicit delete-marker version

Keep explicit marker removals on the canonical version-delete path so
same-key batches are not truncated and pending replication does not retain
source markers. Preserve marker identity and timestamps in the response.

Add real four-disk regressions for mixed version cleanup, duplicate marker
IDs, pending replication, and null/UUID marker removal.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(filemeta): index explicit delete-marker batches

Reuse the batch index and final compaction for ordinary marker removals.
Keep small marker-only histories on the scalar path and preserve fallback
for free versions and complex mutations.

Add scalar-equivalence, boundary, corruption and deterministic scan-count
coverage, a real-disk mixed-history regression, and reproducible marker
benchmark cases. Preserve existing benchmark defaults while honoring CLI
sampling overrides.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(filemeta): remove redundant marker fixture clone

Move the original metadata into its final test fixture after the last
borrow, avoiding the redundant clone reported by strict Clippy checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): validate explicit marker deletion and recovery

Exercise UUID/null and duplicate explicit marker removals over real HTTP
with retries disabled, including immediate four-node visibility and exact
retained payloads. Cover all-drive PREPARED/APPLIED/DECISION process
termination, rollback/commit recovery, and old-ID retries across a second
restart without deleting a fresh marker.

Use flat sampling and fixed per-case fixture batches in the metadata
benchmark. Update profile selection digests for the added regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): synchronize crash fixture startup and flush readiness

Give disk setup and the durable flush boundary separate readiness gates,
keeping each phase bounded to ten seconds. Preserve the real SIGKILL and
restart assertions while reporting early child exits and retaining output.

Reap fixture children on assertion failure and cover early-exit diagnostics.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-24 16:34:31 +09:00
Chris 8c7b3996cb feat(scanner): extract OTLP histogram and counter windows (#8862)
* test(scanner): validate OTLP histogram measurement windows

* feat(scanner): extract exact integer counter windows

* test(ecstore): drain guarded delete fixture rename tails

* test(ilm): retain transition timeout evidence
2026-10-24 16:33:03 +09:00
Chris 68a5b2c89c fix(heal): measure admission lock wait and hold separately (#8863)
* fix(heal): measure admission lock wait and hold separately

* fix(heal): exclude force-start cancellation from lock hold

* test(ecstore): drain guarded delete fixture rename tails
2026-10-24 16:17:41 +09:00
Chris 7f1795f275 test(scanner): add recoverable benchmark service control (#8868) 2026-10-24 14:58:44 +09:00
Chris 3092792c3c fix(scanner): recover an interleaved pause backlog wedge after expansion (#8867)
* fix(scanner): roll a wedged pause backlog forward to its proven ledger

For rustfs/backlog#2873 defect E.

Two interleaved writers could leave a replica state that no claim could
leave. Writer B read writer A's partial commit and selected the stable
rollback point. A's commit then completed and A verified it. B's rollback
stabilization overwrote every member it had read after the commit, and A's
stabilization reached only the remaining member. No commit survived and the
stable fields differed, so every claim and every decommission retirement
failed with "neither a surviving membership commit nor a stable rollback
point".

- When no commit survives and stable consensus fails, select the unique
  newest stable ledger by (writer epoch, generation). A stable field is
  written only after its ledger was selected as authority, so that ledger is
  the last proven one. The next claim stabilizes it and advances normally.
- Keep failing closed for an invalid or unreadable replica, for two
  different stable ledgers at the newest key, and for a missing replica that
  a stored commit names as a member.
- A rollback proof written without an authoritative commit names only
  members that already store a record, so a failed first write to a new
  member cannot turn it into a named member that reads as lost.
- Retirement refuses a rolled-forward ledger until the writer has
  stabilized it, as a retryable handoff conflict rather than a terminal
  authority conflict.
- Tests replay the interleaving on real storage: it converges and retires,
  it converges after another expansion with a failed first write to a new
  member, and a same-key fork or a lost named member still refuses.

* fix(scanner): let retirement stabilize a rolled-forward pause backlog

Follow-up to the defect E roll-forward for rustfs/backlog#2873.

- Retirement no longer waits for a scanner writer. When no commit
  survives and the native selector over every read member (survivors,
  retiring source and queued pools) rolls forward, the planner commits and
  stabilizes that ledger on the survivors under the writers' object fence.
  The previous retryable handoff could exhaust its ~13.5 s budget long
  before a writer re-claimed, and failed the decommission terminally.
- A writer's roll-forward refuses when a stored commit names a member that
  is missing or outside its read, such as a retiring source excluded from
  the writable set. Retirement, which reads that member, decides instead.
- A writer stabilizing a rolled-forward ledger writes missing members
  first and stops at the first failure. A member that only read as missing
  rejects the create, so no surviving proof can omit its newer record.
- Docs describe the actual roll-forward preconditions and the retirement
  exit.
- Tests: retirement converges the wedge with no writer; a writer refuses to
  roll forward past an unread source that holds the newest ledger and
  retirement keeps it; a spurious missing read changes no record and the
  next claim converges on the newest ledger.

* fix(decommission): create missing pause backlog survivors first

Follow-up to the defect E retirement roll-forward for rustfs/backlog#2873.

- Retirement creates survivors that read as missing one at a time, before
  any stored survivor, and stops at the first failed create. A transient
  create failure on a new pool's set no longer leaves every other survivor
  holding a commit that names a still-missing member, which no later
  selection could leave.
- A rolled-forward retirement first seeds a rollback proof that names only
  the survivors that store a record; the commit still names every survivor.
- Retirement rolls forward only when the survivors cannot select a ledger
  on their own or select the same one. Survivors that agree on a different
  ledger keep the source-authority check.
- The scanner writer applies the same missing-first, stop-at-first-failure
  order to every stabilization with a missing member, including a seed from
  an old commit selected as the rollback proof.
- Tests: a failed new-pool create during retirement retries to convergence;
  a spurious missing read cannot seed an old commit over a newer member;
  the rolled-forward precedence over source stable records is pinned.

* docs(decommission): state the all-missing survivor limit of backlog roll-forward
2026-10-24 14:52:30 +09:00
Chris e542e62f07 feat(heal): record namespace lock acquisition waits (#8866) 2026-10-24 14:49:59 +09:00
Chris d90419f4dc feat(scanner): count cache save payload bytes per attempt (#8865) 2026-10-24 14:23:02 +09:00
Chris bbe2f5a3d3 fix(ecstore): skip orphan cleanup for leading-slash listings (#8864) 2026-10-24 13:22:06 +09:00
Chris b5f275ca3f fix(scanner): collect benchmark metrics from realtime route (#8859)
* fix(scanner): collect benchmark metrics from realtime route

* docs(scanner): align benchmark help with realtime route

* test(e2e): bound parallel seeding for scanner usage checks
2026-10-24 12:45:42 +09:00
Chris c2e6b974e1 test(e2e): bound parallel seeding for scanner usage checks (#8861) 2026-10-24 12:44:45 +09:00
Chris b39fcb2e85 test(heal): cover rc.5 upgrades and refresh ctutils (#8847)
* test(e2e): use the resumed decommission completion budget

* test(heal): exercise checkpoints written by rc.5

* test(heal): route upgrade fixtures through the storage facade

* fix(deps): update ctutils to address constant-time advisory

* test(lifecycle): gate recovery I/O while advancing test time
2026-10-24 11:20:05 +09:00
overtrue 80fff4b5c5 Merge gateway/integration (#8844) into the main sync
Only the s3s footprint baseline conflicted; s3_error_lines tightens to 289.
2026-10-24 09:30:22 +08:00
Chris 301e3a9f34 fix(decommission): count settled entries once, back off repeat pauses (#8848)
* fix(decommission): count settled entries once and back off repeated pauses

Follow-up to rustfs/rustfs#8824 for rustfs/backlog#2873 defect F.

- Count an entry's versions when the entry settles (copied and cleaned up,
  or retained) instead of as each version copies. A partially moved entry
  that is deferred, paused and resumed, or interrupted, no longer counts its
  already-moved versions again when the resumed pass finds them on the
  target.
- Hold the watchdog relaunch of a decommission that pauses again on the
  same cause without progress, doubling from 30 s up to 15 min, and warn with
  the cause. A first pause still relaunches on the next pass; the pause never
  becomes a terminal failure.
- Log each drive's error when a delete version loses write quorum, so the
  cause of a sub-quorum delete-marker copy is recorded instead of only its
  rollback.
- Tests: pause/resume counting, a target whose metadata is below read
  quorum, the copy-quorum pause cause, the relaunch backoff, and a rollback
  test that asserts the restore attempt instead of a tracing log.

* fix(decommission): measure pause progress by cleaned entries

Review follow-up on the pause/resume counting and relaunch backoff.

- A pause now carries this generation's progress counters into the
  pool metadata it persists, like a progress checkpoint. Before, everything
  counted since the last checkpoint was dropped at the pause.
- A retained entry (one whose source stays) is relisted by every resumed
  pass. A per-instance ledger counts its versions once per decommission
  generation, so retained versions no longer grow per round in one process.
- The relaunch backoff measures progress by entries moved off the pool and
  completed buckets, which a relisted retained entry cannot advance. Before,
  the recounted retained entry looked like progress and reset the backoff.
- Tests: exact totals across a pause and a resume from persisted metadata;
  a target kept below quorum defers the entry with nothing counted; a
  retained entry plus a repeated cause escalates the backoff over three
  pauses, with the watchdog path relaunching a first pause and holding a
  repeat.

* fix(ecstore): bound retained decommission accounting
2026-10-24 10:11:44 +09:00
Chris 6535c28f1a fix(scanner): recover backlog across interrupted pool expansions (#8858)
fix(scanner): recover stable membership after repeated expansion
2026-10-24 10:01:10 +09:00
Chris d91dfcedd2 refactor(s3): migrate private multipart validation errors (#8844) 2026-10-24 09:17:48 +09:00
overtrue 79c4fedf8c Merge gateway/integration into the main sync
Brings in #8825, #8831, #8836 and #8839. Conflicts were import lists and test blocks (both sides kept), plus the new restore_copyback request flag next to main's data-movement field. Main replaced completed_multipart_object_part with prepare_complete_multipart_parts, so the stale import is dropped.

Main's xl.meta fast-path scanner (inspect_v2_object_slice) predates #8836 and classified a transitioned version with a pending v1 restore copyback as having no physical data dir, which made find_unshared_data_dir_for_version drop the copyback data dir. The scanner now reads the copyback marker with get_consistent_bytes semantics (both aliases, ASCII case-insensitive, empty or conflicting values count as absent), and uses_data_dir keeps main's allocation-free transition check plus #8836's copyback exclusion. find_unshared_data_dir_for_version_keeps_pending_copyback_data_dir failed on the plain merge and passes now.
2026-10-24 07:51:55 +08:00
Chris 76f4342ae2 feat(scanner): observe canonical heal task terminals (#8851) 2026-10-24 08:50:13 +09:00
Chris 73c846e09c refactor(storage): migrate option errors to domain types (#8839) 2026-10-24 05:50:10 +09:00
Hauser 128634f102 fix(notify): preserve marker IDs and reduce event allocations (#8853)
* fix(notify): preserve delete marker version identities

Use the committed marker identity for batch creation notifications and map
null versions at the notification boundary. Keep explicit marker removals
and synthetic directory identities consistent with the S3 response.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(notify): remove redundant delete version fallback

Single-object storage results already normalize absent version IDs to nil
UUIDs for versioned and suspended buckets. Keep the existing response
projection and reserve marker fallback handling for batch results.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(notify): avoid redundant delete and event allocations

Skip successful response projection for quiet batches and move committed
names directly into lightweight notification data. Traverse committed
results through the single-object cache invalidator, removing the unused
batch wrapper and temporary key copies.

Consume owned event fields and use borrowed response-ID entries while
preserving version selection, metadata filtering, and dispatch semantics.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-24 04:14:26 +09:00
Chris abe14e2f97 feat(scanner): configure startup recovery backoff cap (#8845)
* feat(scanner): configure startup recovery backoff cap

* test(scanner): compare recovery retry count without narrowing

* docs(scanner): document recovery backoff cap and expose it in Helm

List scanner.recovery_backoff_cap in the runtime controls table, drop the
fixed ten-minute wording from the follower recovery section, and map
config.rustfs.scanner.recovery_backoff_cap_secs to
RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS in the chart.
2026-10-24 03:09:39 +09:00
Hauser 7d6172e666 chore(deps): refresh crates and upgrade hotpath to 0.29.0 (#8852)
Run two cargo update and cargo upgrade rounds from origin/main.
Upgrade the hotpath crate family to 0.29.0 and refresh compatible
AWS, Redis, profiling, crypto, UUID, and certificate dependencies.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-24 03:09:34 +09:00
Chris 6938c467d0 fix(ecstore): finalize restore metadata after local copy-back (#8836)
* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback
2026-10-24 02:34:29 +09:00
Chris 3254e79efd perf(scanner): transfer owned caches without redundant clones (#8849) 2026-10-24 00:49:07 +08:00
Chris da7dc74147 fix(quota): retire degraded floors from durable usage reports (#8832)
* fix(quota): retire degraded floors from durable usage reports

* docs(quota): clarify retained usage cache refresh behavior

* fix(admin): report unknown bucket usage as null in GetBucketQuota

GetBucketQuota serialized missing usage as size 0, which contradicted the
D3 rule that unknown usage is never reported as zero and, after retiring
degraded floors, made a bucket near its limit look empty. Make size
optional and stop collapsing a missing or unreadable usage snapshot to 0.
2026-10-24 01:46:02 +09:00
Chris db55098aa6 fix(scanner): recover dual corruption with retained epoch fencing (#8833)
* fix(scanner): recover dual corruption with retained epoch fencing

* fix(scanner): preserve recovery markers and legacy quota floors

* test(scanner): expect retained usage recovery marker state

* docs(scanner): register dual-corruption recovery evidence objects

Add the .bloomcycle.bin.dual-recovery/ evidence family to the persisted
objects table and describe its owner, write-once lifecycle and recovery
rule in Deletion And Recovery Rules.
2026-10-24 01:43:59 +09:00
Chris 47e1850d7e fix(scanner): preserve remote checkpoint coverage identity (#8827)
* fix(scanner): preserve remote checkpoint coverage identity

* test(scanner): verify remote plan identity propagation

* test(rpc): disambiguate scanner capability body collection
2026-10-24 00:22:49 +09:00
Chris 4e88ebcf30 fix(heal): preserve typed replacement readiness deferrals (#8823)
* fix(heal): preserve typed replacement readiness deferrals

* docs(heal): align retry compatibility markers with guards
2026-10-24 00:22:41 +09:00
Ramakrishna Chilaka cfee2ffaba perf(ecstore): bound xl.meta read workers and enable HEAD early-stop (#8846)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
2026-10-23 22:32:48 +08:00
Chris 623601d2f9 fix(test): select the current G09 upgrade case (#8841) 2026-10-23 23:04:59 +09:00
Chris 2474adc50f feat(heal): inspect blocked checkpoint validation without mutation (#8838) 2026-10-23 23:03:57 +09:00
Hauser c8cb1d86d3 fix(scanner): streamline budgeted metadata walks (#8837)
* fix(scanner): budget directory enumeration I/O

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(scanner): skip redundant metadata walk work

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(scanner): share budgeted directory traversal

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-23 21:07:09 +09:00
Hauser 607393e6fa fix(ecstore): unify multipart generation fence lifetimes (#8830)
* fix(s3): expose typed causes for unavailable responses

Keep the existing S3 retry contract while distinguishing namespace lock
timeouts, other lock refusals, and storage write quorum failures. Reuse one
bounded typed-source walk and document multipart session attribution.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): preserve XML type on attributed errors

Keep application/xml when the pinned serializer replaces generated headers
with the custom cause map. Assert the media type on direct and wrapped lock
timeouts and namespace/write-quorum error responses.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): unify multipart generation fence lifetimes

Own registry registration before asynchronous incarnation validation so
cancellation cannot preserve a memo beyond real lifecycle-lock coverage.
Reuse successful validation for overlapping PUT and multipart operations,
and retain the full owner through detached part commits and orphan cleanup.

Keep cold validation concurrent and preserve authoritative snapshot,
borrowed-fence, and disabled-lock checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): remove redundant pending closure

Pass std::future::pending directly to the incarnation validator in the
cancellation regression test so strict Clippy accepts the test target.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-23 20:14:34 +09:00
Chris 5323df7049 fix(protocols): clear clippy failures after T1.5 merge (#8831)
fix(protocols): clear clippy failures after merge
2026-10-23 17:37:26 +09:00
Chris daa9529fc6 fix(decommission): keep partially moved versions readable (#8824)
* fix(decommission): keep partially moved versions readable and defer lost-quorum copies

A version copy whose target write kept losing drive quorum was counted as a
permanent per-object failure, and a partially moved entry made every copied
version unreadable: the retained source lists all versions while the target
lists only the copied ones, and the cross-pool identity check treated that
version-list difference as a same-time conflict (SlowDownRead).

- Compare the version-list fields (num_versions, successor_mod_time,
  is_latest) only for exact identities; data-movement equivalent copies of a
  version stay readable while the entry is partially moved.
- Defer a delete marker, object or tiered version copy that exhausts its
  in-place retries on a drive quorum error to the end-of-round replay instead
  of counting it failed; a persistent loss pauses the decommission, which the
  local decommission watchdog resumes.
- Skip the backup rename when rolling back a delete transaction whose journal
  recorded no original metadata, so a marker created on a missing key no
  longer logs a NotFound that reads as the cause of the failed write.

Refs rustfs/backlog#2873

* fix(ecstore): close copy quorum and tiered read gaps

* test(ecstore): synchronize capacity fixtures by object
2026-10-23 16:33:16 +09:00
Chris 55b6fbd260 refactor(protocols): move StorageBackend to gateway types (#8825)
* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests
2026-10-23 15:46:28 +09:00
Chris 1820a4c40b fix(heal): list local blocks and quiet legacy ambiguity logs (#8826)
* fix(heal): list local blocks and quiet legacy ambiguity logs

* test(heal): cover storage and scheduler ambiguity logs
2026-10-23 13:40:00 +08:00
Chris 631cd0ceca fix(ecstore): preserve migrated reads and retry busy pool heals (#8821) 2026-10-23 12:09:38 +09:00
Hauser 5778384b68 chore(deps): refresh ICU and profiling dependencies (#8820)
Update Pyroscope to 2.1.4 and compatible ICU, yoke, and zerovec crates.
Cover normalized Unicode hostnames and loopback rejection in egress tests.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-23 11:03:04 +08:00
overtrue dd677cc121 Merge main into gateway/integration
Picks up main through 44a73b0be. Conflicts: rustfs/src/error.rs keeps the rustfs-s3-types ApiError conversion and drops the helper main's #8816 replaced; scripts/check_s3s_footprint.sh keeps the integration script, whose baseline lives in .config (main only retuned a constant the integration script does not have). Main's new tests use the rustfs-s3-types error code in admin replication and reach s3s::BodySizeLimitExceeded through the app's test-only s3 facade, so the s3s file count stays at 118; s3_error_lines tightens to 302. The test-count baseline is regenerated; the e2e selections are unchanged.
2026-10-23 08:40:02 +08:00
overtrue 0c2f327974 Merge gateway/integration into the main sync
Brings in #8812 and #8818 from gateway/integration. #8812 predates main's heal error classifier (#8778); its two retryable variants (TooManyOpenFiles, OutdatedXLMeta) move into the classifier's transient arms for storage and disk errors, and its test is kept.
2026-10-23 07:22:20 +08:00
overtrue df42cf1ce8 test(e2e): use a non-default region for the differential negative control
Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).
2026-10-23 07:21:19 +08:00
Chris d290d46b34 test: reserve nextest capacity for progress probes (#8818) 2026-10-23 06:12:31 +09:00
Hauser 44a73b0bef fix(s3): preserve typed upload errors across body readers (#8816)
* fix(s3): report payload SHA256 mismatches accurately

Distinguish signed payload SHA256 failures from MD5 and additional
checksum failures at the shared API error boundary. Preserve the
payload error through archive extraction and use typed s3s errors.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(s3): classify body failures in one bounded pass

Share typed body-error classification across direct and storage I/O.
Retain marker precedence while replacing repeated source-chain scans,
and map storage-wrapped short HashReader bodies to IncompleteBody.

Cover nested priorities, real short reads, and one traversal budget.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(s3): route upload stream errors through the app facade

Keep the real stream-error fixtures behind the existing test-only S3
boundary without increasing the direct s3s import footprint.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): preserve typed body failures across archive reads

Route TAR entry failures through the shared archive error adapter and
preserve every classified body error. Keep generic archive parsing as
InvalidArgument without formatting discarded transport diagnostics.

Use the existing timeout marker for PUT and Snowball body guards. Add
regressions for real stream verification after sniffing and timeout
responses, and tighten the reduced S3 macro footprint baseline.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-23 05:07:43 +09:00
唐小鸭 91007466c9 fix(site-replication): keep stored name on endpoint-only edits (#8817)
An edit that omits the name, as mc admin replicate update --endpoint
does, was normalized with a name inferred from the new endpoint host and
overwrote the stored peer, renaming the site on every peer map while its
own state name stayed unchanged.

update_peer now keeps the stored name for a known deployment ID when the
edit carries none, covering the public edit, the endpoint-refresh commit
and internal peer deliveries. The public handler also fills the stored
name before planning so the fan-out and refresh journal carry it to
receivers running older versions. Add and join of unknown peers still
infer the name from the endpoint.
2026-10-23 05:06:54 +09:00
唐小鸭 6e3484916f fix(replication): reject connection edits on site-replication targets (#8814)
A set-remote-target update with the creds group on a bucket target that
belongs to a site-replication peer skipped the connection change, logged a
warning, and still returned 200. The Console's bucket replication form sends
exactly that request when an endpoint changes, so operators saw success while
the stored endpoint, credentials, and TLS settings stayed unchanged.

Return 400 InvalidRequest instead, naming the target and the evidence that
makes it peer-owned, and pointing at a site-replication edit. The ownership
probes are unchanged; the decision is extracted so it can be unit tested.

Document how to change a site's address: the local endpoint is recorded once
at setup and never re-derived, edits must reach every other peer at its stored
endpoint, so sites move one at a time (or old addresses are restored
temporarily), followed by a resync. Commands use rc because mc's resync does
not work against RustFS and its update renames the site.
2026-10-23 02:29:13 +09:00
唐小鸭 e1e710cb5d fix(admin): serve bare madmin info bodies on the MinIO prefix (#8815)
GET /v3/info and /v3/storageinfo have wrapped the madmin InfoMessage and
StorageInfo under an `info` key since the cluster snapshot view (#3803).
madmin decodes these bodies directly, so `mc admin info` printed empty
values and `mc admin replicate resync start` rejected every alias because
the local deploymentID read as empty.

Requests on the MinIO-compatible /minio/admin prefix now receive the bare
madmin body, following the existing is_compat_admin_request convention.
The /rustfs/admin prefix keeps the envelope that rc (which requires the
`info` key), the console, e2e tests, and auto-testing scripts read.
2026-10-23 02:25:32 +09:00
Chris d9ab8760ee test(e2e): accept retried rebalance deferrals; retry busy decom preflight (#8813)
test(e2e): accept retried rebalance deferrals and retry busy decommission preflight
2026-10-23 02:10:04 +09:00
Chris 0d297ed8cd fix(heal): retry transient metadata conflicts (#8812)
* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline
2026-10-23 01:47:55 +09:00
Ramakrishna Chilaka 67442a5ba3 perf(filemeta): fast-path canonical V3 header matching and borrowed version merges (#8811) 2026-10-23 00:39:23 +09:00
Hauser df10dc7864 fix(scanner): fence backlog writes against stale pool membership (#8799)
* perf(s3): borrow options during delete preflight

Pass the existing request options by reference to the metadata preflight instead of cloning them solely for that borrow. Keep the getter's mutable snapshot copy and the later owned-delete copy.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): fence backlog writes against stale pool membership

Reject native scanner publications unless their captured writable pool cohort
matches the fenced durable topology. A smaller startup view must not replace
an expanded membership commit with its older stable rollback.

Keep single-replica test injection fenced by the complete current cohort and
cover rejected startup claims, unchanged CAS revisions, and recovery of the
acknowledged expanded commit.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): use current cohort for healthy scanner CAS writes

Keep the writer's cached topology stale while explicitly capturing the
surviving membership for healthy-target publication and CAS checks. Also
assert that the old cohort rejects healthy-target writes without publishing.

Preserve retiring-source rejection, payload and ETag checks, and canceled
source membership repair without changing production admission.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 23:17:15 +08:00
Ramakrishna Chilaka 5c463ff8d3 perf(filemeta): avoid per-replica cloning during metacache quorum resolution (#8810) 2026-10-22 22:01:58 +09:00
overtrue b8dffa2f1d Merge main into gateway/integration
Picks up the 20 main commits since 62b6f5f3f. The one conflict is the tier free-version recovery test setup: it keeps the integration-only fixture isolation (background recovery disabled while the fixture starts) and main's mock tier registration (#8804 skips scans with no tier). The e2e selections are unchanged by these commits; the test-count baseline is regenerated.
2026-10-22 20:17:09 +08:00
Hauser 1e36e18265 feat(metrics): classify terminal remote walk failures (#8809)
Record a bounded failure boundary for each non-successful logical remote
walk while preserving the existing outcome, error source and retry behavior.
Distinguish stream open/read, downstream write/flush, wrapper admission and
deadline failures, and unfinished operations dropped before completion.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 20:12:51 +08:00
cxymds 0b25604503 fix(config): preserve persisted compatibility fields (#8803)
* fix(config): preserve persisted compatibility fields

Bound ignored configuration diagnostics, retain unsupported aliases and KVS attributes during saves, and prevent stale shadow values from resurfacing. Add a pinned MinIO fixture and regression coverage for decode, reset, encryption, and logging behavior.

* fix(config): satisfy ecstore clippy

* test(config): ignore JSON object key order in roundtrip

* fix(log-analyzer): recognize persisted config decode failures
2026-10-22 20:05:21 +08:00
Hauser 89c3282ae2 test(scanner): reserve capacity for all native backlog fixtures (#8802)
Include single-pool, status, and expansion fixtures in the existing
native scanner reservation without changing budgets or assertions.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 17:54:05 +08:00
Hauser 0d0e6d8497 fix(ecstore): preserve scanner authority during decommission (#8807)
Fence native backlog writes against their captured pool membership and retain
queued replicas as read-only proof members during retirement. Stabilize the
topology test workload and bound pre-ACK recovery separately.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 17:42:20 +08:00
Ramakrishna Chilaka f1a0263ab1 perf(utils,filemeta): vectorize path and metadata key hot loops with SWAR (#8806) 2026-10-22 17:40:07 +08:00
Hauser bea4be68a2 chore(deps): upgrade workspace parking_lot to 0.13 (#8801)
* chore(deps): upgrade workspace parking_lot to 0.13

Move all six direct workspace consumers to parking_lot 0.13 while
retaining the 0.12 dependency line required by upstream crates.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(connect): stabilize client failure classification

Use an ephemeral loopback listener that explicitly rejects proxy CONNECT
and direct TLS connection establishment instead of assuming fixed ports
will promptly refuse connections. Give classification its own bounded
transport and measurement budget while retaining the timeout tests.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 17:35:02 +08:00
Hauser 90a39a5083 perf(heal): borrow intents during retirement proof checks (#8800)
Share the existing canonical identity resolver with a non-authoritative source eligibility query. Group borrowed intents and create owned anchors only for retired generations, preserving cache and responsibility contracts and moving proof keys.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 17:22:37 +09:00
Hauser 11381c260f fix(ecstore): restore marker write quorum on decommission resume (#8793)
* fix(ecstore): restore marker write quorum on decommission resume

Verify equivalent target markers at write quorum and republish partial
copies before completion. Preserve exact marker creation semantics and
legacy birth identities while retaining ordinary delete and purge guards.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(odm): run delete marker regression on a bounded stack

Construct and run the deep real-storage future on the existing large-stack
test worker so the default libtest stack does not overflow. Preserve the
registered test name, serial guard, and single-part/multipart assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): use marker majority for resumed copies

Confirm existing and republished delete markers at the configured-set
majority used by their delete transaction instead of the EC data quorum.
For a twelve-drive EC4 set this accepts seven equivalent markers while
still rejecting six, including when the online inventory is shortened.

Cover named/null markers on a real rebalance retry with retained source
history. The regression fails with InsufficientReadQuorum before the fix.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): keep catch-up fixture timestamps consistent

Admit the seeded catch-up using its observation timestamp so a durable
writer claim crossing a wall-clock second cannot spuriously rate-limit it.
Model an older claim timestamp to exercise that boundary deterministically.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 17:20:56 +09:00
cxymds 89b98d65c2 fix(ecstore): skip empty tier recovery scans (#8804) 2026-10-22 15:53:50 +08:00
Hauser 335612f452 perf(heal): skip replay anchor clones for empty repair queues (#8798)
Use a conservative work hint for verified and typed legacy events, retaining the original capacity-maintenance pass and metrics publication. Preserve durable identities, journal retention, retries, and shutdown behavior, with isolated real-runtime and queue lifecycle regressions.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 16:38:06 +09:00
Ramakrishna Chilaka 0bade0ca3e perf(ecstore,filemeta): batch listings and buffer metacache streams (#8791)
* perf(ecstore,filemeta): batch local walk_dir metadata reads and buffer metacache streams

* fix(ecstore): stream completed listing metadata reads

Publish each result before starting the next blocking read, preserving
per-object progress and cancellation without a timed chunk queue.
Keep dispatch batching, byte limits, and namespace/error stops intact.
Cover multiple batches, page boundaries, and oversized-envelope resumption.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore,filemeta): verify production listing streams

Replace the simulated directory-walk benchmark with the public local disk
path. Validate fixture output before timing and retain metacache-only
stream benchmarks. Pin fragmented reads, read coalescing, and canonical
UUID namespace probing against the reference behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): route listing benchmark through storage API

Centralize local listing benchmark imports in the shared benchmark
storage API boundary required by the architecture migration rules.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): keep catch-up fixture timestamps consistent

Admit the seeded catch-up using its observation timestamp so a durable
writer claim crossing a wall-clock second cannot spuriously rate-limit it.
Model an older claim timestamp to exercise that boundary deterministically.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 14:54:34 +08:00
Hauser 5749752217 chore(deps): refresh cloud SDKs, runtime and profiling (#8787)
* chore(deps): refresh GCS, profiling and allocator dependencies

Update published GCS, Pyroscope, mimalloc, and PPMd crate versions.
Align the pinned erasure-fixture test recursion limit with ECStore
to compile GCS-enabled test targets.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): update AWS SDKs, Tokio and memmap2

Refresh compatible AWS service and Smithy dependencies, Tokio timers,
and memory mapping support with the regenerated lockfile.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 12:17:47 +08:00
Hauser 03c65bcba8 fix(ecstore): identify scanner retirement planner failure phases (#8796)
* fix(ecstore): identify scanner retirement planner failure phases

Bind the existing native planner error context to each retirement phase while preserving typed retry classification and all storage fences. Exercise the actual wrapper in an isolated process across unavailable and typed failures.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): type the unavailable retirement planner error

Replace the new formatted other error with a private typed error so the error-construction ratchet remains intact. Preserve all phase messages and the existing native retry classification.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): cover unavailable planner error identity

Check the private unavailable cause for every phase through the existing IO boundary. Retain native handoff classification and verify same-phase quorum bucketing after fallible disk narrowing.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 12:17:16 +08:00
Hauser 2feb09b8eb test(obs): cover contended identity under netlink denial (#8797)
* test(obs): cover contended identity under netlink denial

Check that the production identity entry stays nonblocking while a name
writer is held and real netlink discovery is denied, then observes the
published name after release. Release the writer before joining so a
blocking regression reports a bounded failure.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(obs): keep identity contention behind common helpers

Move the held-writer fixture behind a test-only common helper and enable
it through the Linux x86_64 obs dev-dependency. Preserve the timeout,
writer release before reader join, and identity restoration ordering.
Exercise callback completion and unwind cleanup in the common regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 12:09:53 +08:00
Hauser be8357f815 feat(metrics): observe HTTP entry-to-dispatch latency (#8788)
Record the first dispatch or undispatched termination at the existing
external request boundary, with entry-gated timing and owned task scope.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 11:27:53 +08:00
Hauser 8eb67ac38c perf(heal): reuse the verified repair proof queue (#8790)
* perf(heal): reuse the verified repair proof queue

Retain proof events in place under the existing registry lock and keep
the pre-consumption capacity target and exact identity matching.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(common): move the final proof mismatch fixture

Avoid a redundant clone in the last owned mismatch fixture.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 11:12:20 +08:00
Hauser a47649fb60 feat(metrics): expose allocator reclaim statistic totals (#8782)
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 09:15:15 +08:00
overtrue 62b6f5f3fc Merge main into gateway/integration
The scheduled sync has stopped on conflicts since 2026-10-19. Resolved by keeping both sides: main's restructured delete-rollback path in disk/local.rs with the integration-only native-publication record, main's per-instance region with the rustfs-s3-types Region, main's new lifecycle, rebalance and admin code ported to the persisted lifecycle types and rustfs-s3-types errors, both sides' new scanner tests, and the union of the access_consumer imports. The e2e smoke, full (darwin) and repl-nightly selections and the test-count baseline are regenerated from the merged tree.
2026-10-22 10:14:41 +09:00
Ramakrishna Chilaka c0f3f9a108 perf(filemeta,ecstore): index metadata batches and trim decodes (#8784)
* perf(filemeta,ecstore): avoid redundant decodes and clones on xl.meta write and delete paths

* perf(filemeta): remove restore parsing allocations and add guards

Borrow restore header fields instead of allocating temporary vectors and
reuse the canonical dual-prefix metadata key guard. Cover scanner/full
decoder equivalence on historical and mutated records, restore grammar,
and the complete old-current observation tuple. Add the missing Criterion
entry for unshared data-directory selection.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* bench(filemeta): isolate metadata mutation and batch deletion costs

Move the metadata-only Criterion target to filemeta so it does not build
ECStore. Add deterministic physical and inline dense/sparse deletion
cases with the same fixtures available to baseline and candidate builds.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(filemeta): index ordinary metadata deletion batches

Use a batch-local version lookup, compact directory summaries, and lazy
physical reference counts for larger ordinary V2 deletion batches. Decode
each indexed body at most once and compact the version vector once.

Flush pending removals before canonical fallback and preserve per-request
validation, partial failures, inline-key aliases, and cleanup decisions.
Keep small batches and transition/legacy/ambiguous records on the scalar
path. Cover indexed/scalar state equivalence and fail-closed cache errors.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(filemeta): enforce linear batch decoding and safe fallbacks

Count V2 scans in tests to guard the indexed path's linear work contract.
Exercise marker, retention, restore, transition, free-version, and pending
purge requests against the canonical sequential path. Compare logical
state and deterministic headers for re-encoded maps while retaining exact
byte equality for ordinary indexed deletion and opaque metadata.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 09:14:14 +08:00
Hauser 956395a61b fix(diagnostics): preserve context across task handoffs (#8785)
Capture existing diagnostic operation, caller and first-entry context at bucket, background and tracked-mutation handoffs. Keep absent context absent and retain deferred work and admission ownership through the established storage facades.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 09:14:01 +08:00
Hauser 4597bb91a8 feat(metrics): count scanner wakeups and cycle attempts (#8786)
Expose selected outer-wait reasons separately from runner-call attempts.
Preserve scanner scheduling, pause, rate-limit and budget boundaries.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 09:13:37 +08:00
Hauser 87caf45cfe perf(s3): remove unused multi-delete result clones (#8789)
Keep the original storage results and all required request/replication
ownership while dropping two bindings with no consumers.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 09:13:21 +08:00
Hauser 15b7040e78 perf(scanner): move owned usage snapshot keys (#8792)
Consume completed aggregation entries at their terminal conversion so
bucket and replication target keys can move into the output maps.
Preserve validation, rejection and publication behavior.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 09:13:06 +08:00
Chris ffd94cfe1b fix(decommission): finish multipart null versions superseded on target (#8774)
* fix(decommission): finish multipart null versions superseded on target

A decommission entry for an unversioned multipart source whose key
already holds a newer null version on the target settles as superseded
and aborts its staged upload. The cleanup's target proof saw the newer
object, could not match it to the staged identity, and failed closed
with a capacity-blocked error, so the routine paused, the source was
never cleaned up, and the error read like a minimum-free-drive block.

The staged upload is already proven absent and the target lookup is
exact: a different version id, or a null target with another
modification time (one null slot per key), is a different object, so
the staged copy holds nothing on the target and the pending intent is
cleared. Only a versioned target with the staged version id but another
modification time stays fail-closed. The remaining identity-proof
failures use a typed error without capacity wording.

Add three-pool decommission regressions for restored-transitioned and
plain multipart null sources superseded on the target.

* test(decommission): pin multipart target proof failures as terminal

Document the accepted residual of the null-slot proof: a metadata-only
self-copy keeps a published but uncharged staged copy's data while
moving its modification time, so its marker is released and one
object's bytes stay uncharged.

Add a three-pool routine test where the target already holds an
unowned null object with the source's exact identity: the cleanup's
ownership-proof failure fails the decommission with no capacity-blocked
reason instead of pausing it.
2026-10-22 09:22:24 +09:00
Chris be0986115c refactor(s3select): run queries on gateway DTO request shapes (#8794)
* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.
2026-10-22 08:16:48 +09:00
Chris 831029e5b6 docs(agents): add three-verdict finding contract to security skill (#8795)
Require every security candidate to name the lower-trust principal, the crossed boundary, the affected resource, and the observable result, and to land under exactly one of `confirmed`, `needs_validation`, or `rejected`. Only confirmed findings carry severity; a trace blocked on an unresolved deployment fact is reported as that fact instead of an assumption.
2026-10-22 08:06:34 +09:00
Chris 1bd91c8dad refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)
Part of T1.5 (rustfs/backlog#2746), batch 1.
2026-10-22 06:15:04 +09:00
唐小鸭 1e817524e8 fix(replication): keep replicated multipart null versions as null (#8763)
A versioned multipart completion minted a new version when the staged version id was absent or nil, so an inbound replica of a source null version written by multipart landed under a fresh UUID. Keep the null version for authorized inbound replicas, and address the null version as versionId=null on the worker convergence HEAD like the other target reads.

Refs rustfs/rustfs#8747
2026-10-22 02:46:39 +08:00
Hauser 66bf2c0a75 perf(heal): aggregate MRF lifecycle metrics in one pass (#8773)
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 02:46:26 +08:00
Hauser f06ae3dc34 feat(metrics): observe local walk lifecycle and output bytes (#8775)
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 02:45:58 +08:00
Chris 8af4070d19 fix(decommission): move the free version of an expired tier version (#8777)
* fix(decommission): move the free version of an expired tier version

Decommission only queued the lifecycle expiry of a transitioned version, so a source cleanup that ran first deleted the version itself and appended its tier free version after the entry's last pass. The post-check then failed the decommission until an operator re-ran it.

Apply the expiry of a transitioned version synchronously, as rebalance does. Its free version is on the source before the cleanup preflight, which reports the entry changed, and the retried entry moves the record to a target in the same pass. An expiry that cannot be applied leaves the version live, so it moves like any other version.

Refs rustfs/backlog#2873

* fix(decommission): move a delete-all expired tier version live

Applying a delete-all expiry inline made its prefix delete wait for the bucket lifecycle write lock behind the entry's own read lock until the lock timeout, blocking every write in the bucket meanwhile. Move such a transitioned version live and leave the delete-all to lifecycle on the target.

Cover a genuinely unapplied expiry and a free-version worker that consumes the source record before the entry moves it.

Refs rustfs/backlog#2873
2026-10-22 02:45:47 +08:00
Hauser c1de42af2c fix(heal): type legacy MRF decisions and defer diagnostics (#8778)
* fix(heal): explain unverified legacy heal outcomes

Preserve storage diagnostics in canonical unknown outcomes and explain
legacy payload identity limits through the shared local/peer status path.
Borrow expected receipt identities while preserving all proof checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): keep diagnostics from pausing merged MRF repairs

Require the actual MRF executor before publishing legacy pause events.
An ordinary queued owner can retain a merged durable MRF notice without
having performed the specialized deep check. Preserve its diagnostic
without granting it authority to park the repair responsibility.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): type MRF outcomes and defer status diagnostics

Carry storage-owned legacy observations in process and return a typed
terminal result only after the qualified MRF Deep check. Publish pause
events from that result and exact identities instead of source or text.
Keep object names out of retry classification and compute only the
status diagnostic selected by the canonical execution state.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 02:45:31 +08:00
Hauser 179e58d5ef fix(ecstore): preserve diagnostic context in config tasks (#8779)
Capture an existing ReadVersion caller and earliest entry before spawning config transactions. Restore them with the existing scoped wrapper while retaining default-feature operation propagation and detached task ownership.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 02:45:17 +08:00
Hauser 5f5485e332 fix(s3): return InvalidPart and streamline multipart completion (#8780)
* fix(s3): classify empty multipart completion as InvalidPart

Treat absent per-drive multipart transaction directories as empty listings so
completion reaches the existing InvalidPart validation. Preserve quorum and
transaction recovery, and cover the HTTP and degraded-drive boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): move multipart completion metadata in one pass

Reuse quorum-validated part metadata instead of repeatedly inserting, sorting,
finding and cloning parts. Keep legacy cleanup identities and ordering while
moving the original part allocation into the completion manifest. Compare ETags
by borrowing their normalized values.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 02:45:05 +08:00
Hauser 65f0599c70 feat(metrics): track usage outcomes and trim decommission probes (#8781)
* feat(metrics): expose usage candidate rejection outcomes

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): borrow usage candidate fixtures

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): reserve capacity for decommission probes

Reserve all nextest slots for the two eight-disk decommission fixtures under both default and CI profiles. Keep their internal concurrency, production lock budgets, assertions and zero-retry policy intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): scope prewrite capacity probes to the target pool

After persisting a new capacity intent, refresh only the target pool baseline. Preserve the global admission and finalization scans, lock order, fencing, and accounting formulas. Verify the reduced probe count with real disk metrics and preserve injected snapshot/error behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 02:44:49 +08:00
Hauser d5e037faac fix(ecstore): retain operation in detached mutations (#8776)
Capture the caller operation before the tracked mutation task is spawned. Keep detached completion, owned guards, result errors and panic join failures intact.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-22 00:41:03 +09:00
Chris 97e600ae69 fix(heal): hold unobserved legacy MRF responsibilities (#8772) 2026-10-22 00:04:16 +09:00
Chris dcb22bd4d5 refactor(ecstore): move bucket configs onto gateway persisted types (#8709)
* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test
2026-10-21 23:51:23 +09:00
Chris 84e96512bd fix(ecstore): honor idle drive monitoring opt-out (#8770) 2026-10-21 23:29:41 +09:00
Chris fa17718444 refactor(admin): move error types to rustfs-s3-types (#8440)
* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.
2026-10-21 22:50:27 +09:00
Ramakrishna Chilaka 62aafc6b95 perf(filemeta,ecstore): avoid intermediate clones in FileInfo decode (#8767) 2026-10-21 22:35:34 +09:00
Chris b06923beaf test(e2e): add scanner/heal topology protection matrix (#8705)
* test(e2e): add scanner/heal topology protection matrix

Add the S00 protection baseline for the Scanner/Heal program
(rustfs/backlog#2873, program rustfs/backlog#2872): seven
scanner_heal_topology_ cases in the distributed e2e lane that pin pool
expansion, rebalance and decommission behavior before the runtime changes.

The fixture seeds a deterministic inventory (empty, inline, external,
multipart, special and nested keys; a versioned bucket with an explicit
null version, three history versions and a delete marker per key; an
Object Lock bucket with legal hold and COMPLIANCE retention), records
every version with its requested and listed version id, length and
SHA-256, verifies the manifest through every node, reads physical
placement from xl.meta on the pool drives, and reads the strict
durability mode back from the admin API and the node logs.

Fault cases hold the mover at proven windows through file barriers that
compile only with the e2e-test-hooks feature: before the multipart target
commit and before source cleanup (shared by decommission and rebalance),
plus the existing pool-metadata before_publish cutpoint. An unreached
window fails the case. DistCluster gains capture_logs_under so a case can
read node startup events without a global environment variable.

docs/testing/scanner-heal-topology-gates.md maps the gates to the tests
and records the timeout basis.

* test(e2e): keep the barrier failure message static

The error(format!) ratchet rejects new formatted io::Error::other call
sites in ecstore. The reached observation already records the stage and
object, so the injected failure needs no per-call detail.

* test(e2e): require e2e-test-hooks for topology fault cases

The four scanner_heal_topology_ fault cases inject at server barriers that
exist only with the e2e-test-hooks feature. Against a default server they
waited for the barrier timeout and reported it as not reached. Check the
verified binary's features first so the missing prerequisite fails
immediately and is named.

* ci(e2e): build the distributed lane with e2e-test-hooks

Every other e2e lane builds and runs the fault-test hooks server. The
distributed lane alone used the default server, so the topology fault
cases could never reach their barriers there. Build and run with the same
feature so the receipt matches, and update the documented local commands.
The upgrade cases spawn the pinned previous release directly and do not
compare feature sets.

* test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR

The topology fixture captured node logs, barrier observations and evidence
in a temp dir removed on drop, so the distributed workflow uploaded nothing
for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow
uploads, when no explicit evidence directory is set.

* test(e2e): remove retired pools before restart; race the pre-ACK start

The queue case restarted with decommissioned pools 0 and 1 still on the
command line, which the server rightly refuses. Add
remove_single_node_pools and restart without them, then expect the
remaining pools renumbered to 0 and 1.

The pre-ACK case now races the start request against the barrier. A start
that returns first, or stays in flight past the documented 60 s budget, is
reported as such instead of as an unreached barrier, and the evidence
records the reach latency.

capture_logs_under now overrides a log path assigned from
RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too.

* test(e2e): bound fault-window waits by the movement budget

On Linux CI a busy target capacity gate delayed the first fault object by
about two minutes, past the 120 s wait borrowed from the server's release
poll. Wait for a movement window within the 240 s movement budget instead,
and fail at once when the decommission or rebalance ends without entering
the window, so an uninjected fault is never counted.

* test(e2e): bound status probes while waiting for a fault window

The admin decommission status request can wait on a mover held at a test
barrier until the barrier's own 120 s release timeout. The window wait
issued that request each loop, so a reached barrier went unnoticed until
the server timed it out and failed the decommission, which surfaced as a
false resume failure. Bound each status probe to 5 s and treat a slow probe
as still running.

* test(e2e): retry a busy decommission start

A decommission start that meets a busy movement gate can answer a
retryable 503 instead of waiting. The harness treated any non-proof
failure as fatal. Share one start helper that retries 503 within the
case's movement budget and the fleet-proof 500 for 45 s, and keeps every
other status fatal. The pre-ACK case retries 503 within its window budget.
2026-10-21 22:08:17 +09:00
hector 023f2e7258 ci(pool): finalize decommission removal and verify surviving pools (#8749)
ci(pool): finalize decommission removal and verify surviving pools (rustfs#8726)

Select suite step 10 together with step 9 and pass
--finalize-decommission so the nightly exercises the post-removal
survivor path: pool ids must stay unique, bucket metadata writes must
keep succeeding across a restart, and no node may log 'pool metadata
layout changed'. Step 10 self-skips when decommission is disabled.

Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-21 22:07:58 +09:00
Hauser 9fad6104c4 feat(metrics): observe remote directory walk lifecycles (#8765)
Track active walks, scratch capacity and accepted bytes while preserving the existing 1 MiB stream copy behavior.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-21 20:49:13 +08:00
Chris d2eda195e6 fix(decommission): defer source cleanup quorum failures (#8760)
* fix(decommission): defer source cleanup quorum failures

A quorum failure while deleting the source of an already copied entry
was fatal and marked the whole pool failed, although the copied source
is still in place and an uncommitted delete transaction is settled by
delete recovery. Carry the cleanup error typed and resolve a quorum
failure as Deferred: the entry is replayed at the end of the round, and
a deferral that persists takes the existing capacity-blocked pause path
instead of a terminal failure. Other cleanup errors and quorum errors
from other stages stay terminal.

Add set-level regressions for parent and child keys (`a/b`, `a/b/c`):
parent cleanup leaves every child file untouched, and parent cleanup
reaches quorum while the child is cleaned up or rewritten concurrently.

* test(ecstore): keep source cleanup errors typed in prefix cleanup tests

Return SourceCleanupError from the test helper instead of formatting it
into Error::other, and shrink the error(format!) ratchet baseline for
core/pools.rs, which lost one call site in the cleanup failure change.

* fix(decommission): retry cleanup quorum failures and count replays once

Retry a source cleanup quorum failure in place, up to three attempts
with the existing source-cleanup backoff, before deferring the entry.

Carry the versions an entry already counted on its deferral, so the
end-of-round replay does not add a fully counted entry to
items_decommissioned and bytes_done again.

A deferral that persists pauses the pool through the existing deferred
drain path. When source cleanup caused it, persist a reason led by
"source cleanup blocked" without the carrier error's free-drive wording,
and report waitingReason "source_cleanup" instead of "capacity".

Add a source_cleanup test fault stage and a routine test that drives a
cleanup quorum fault through deferral and replay to completion.

* fix(decommission): wait seconds between cleanup quorum retries

Give the in-place source cleanup quorum retry a fixed floor of one and
then two seconds, with the existing jitter on top, so it can ride out a
drive reconnect or health-check blip instead of retrying within a few
hundred milliseconds. The wait stays bounded and cancel-safe.

Find a source cleanup pause reason through stage wrappers, as the
capacity-blocked check does, and add a routine test where cleanup keeps
losing quorum: the decommission pauses with a source cleanup reason, the
bucket stays queued and the source stays in place.

Document the additive waitingReason value source_cleanup and its
compatibility with older nodes reading the same pool.bin.

* fix(decommission): never retry a lost source cleanup fence

The cleanup retry classifier used is_quorum_error(), which also matches
NamespaceLockQuorumUnavailable. The delete commit fence reports a lost
namespace lock or mutation fence with that variant, so a lost fence was
retried in place and only stopped later at the pre-cleanup fence check.

A lost fence means this worker no longer owns the key; it must stop
without retrying. Retry and defer only drive quorum failures:
ErasureReadQuorum, ErasureWriteQuorum, InsufficientReadQuorum and
InsufficientWriteQuorum.

---------

Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-21 20:30:22 +09:00
唐小鸭 311f27baa1 fix(ecstore): handle Windows trash cleanup deletion races (#8754)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-21 20:02:39 +09:00
Hauser 1ac35917c2 chore(deps): refresh dependencies and patch certificate trust (#8761)
* chore(deps): refresh dependencies and patch certificate trust

Update published dependencies and certificate verification/root data, and
upgrade rustfs-mimalloc to 0.7.0. Retain the known webpki-roots 0.26.11
compatibility-wrapper advisory without adding an audit exception.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* remove hyper dependency

* remove hyper dependency

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-21 16:40:33 +08:00
shurkus 73ff9a87e8 fix(s3): avoid mutation locks for existing buckets (#8745)
fix(s3): answer CreateBucket for existing buckets without locks

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-21 16:12:43 +09:00
Hauser 102cc18767 test(heal): retain unknown same-key repair across replay (#8759)
Cover actual journal and lifecycle checkpoint replay when an unknown historical source coexists with an exact source-bound repair. Reject a pre-restart lease and retain the complete unresolved record after the new repair is discharged and replayed again.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-21 15:02:01 +08:00
Chris 77a36d92a3 fix(decommission): let a failed start roll back its own activation (#8752)
* fix(decommission): let a failed start roll back its own activation

A fresh decommission start whose peer reload failed could not roll back:
the pool metadata stale-write guard from #6604 refuses any save that
clears a persisted active or queued decommission, so the pool stayed
active with no worker until the watchdog or a leader restart picked it
up.

The rollback save now carries the activation that start committed. The
guard exempts a pool only while its persisted record still equals that
activation field for field, so a worker checkpoint or a later start
keeps the rejection. A canceled record is not an active decommission
and is cleared by the rollback as well. Every other save path passes no
activation and is guarded as before.

A peer may already have reloaded the activation and started a worker
before the rollback. Saves are now fenced by decommission generation: a
write carrying a start_time is rejected when the persisted record
belongs to a later start, or has no decommission state and was updated
after that start_time, so such a worker cannot make the pool active
again. Its queued promotion and progress checkpoint also adopt the
persisted record and stop the worker.

Refs rustfs/backlog#2873

* fix(decommission): fence terminal writes of a withdrawn decommission

Terminal records carry no start_time, so the generation fence did not
cover them: after a start rollback, a peer worker of the rolled back
generation that failed (for example on a bucket completion) could still
persist a failed record over the restored pool. The merge now rejects a
failed, canceled or complete record over a persisted record without
decommission state; every attempt persists decommission state first, so
no legitimate path writes a terminal record there.

Peers also drop a withdrawn decommission sooner. A pool metadata reload
replaces a running decommission with a newer record that withdrew it,
even under an active worker; bucket completion adopts such a record like
promotion and progress checkpoints do; and a withdrawn progress
checkpoint stops the worker instead of logging a failed save and moving
more entries.

Tests cover the peer failure mark, a rollback retry that sees a worker
checkpoint land between attempts, and the reload withdrawal. A heal
fixture that wrote complete records over a fresh pool.bin now persists
the active attempt first.

Refs rustfs/backlog#2873

* fix(decommission): fence peer writes after a retry start rollback

A retry start replaces a failed or canceled record that keeps its
unresolved ledger, so its rollback restores that terminal record rather
than clearing the pool. The withdrawal checks only recognized a record
without decommission state, and the generation fence ignored a persisted
record without a start_time. A peer worker of the rolled back retry
could therefore checkpoint the pool back to active.

The generation fence now compares the persisted last_update whenever
the persisted record has no start_time of its own (cleared, terminal or
queued). The progress checkpoint also checks the generation against the
record read under the pool metadata lock, as bucket completion already
does, and the save-path withdrawal check treats a terminal record newer
than the running view as a withdrawal, so the peer adopts it and stops.

A reload still never lets a newer terminal record replace an active
worker's view, so after a retry's rollback a peer worker stops at its
next bucket completion or progress checkpoint rather than on reload.

Tests cover the retry rollback against a peer checkpoint and direct
write, and a peer bucket completion after a fresh start's rollback.

Refs rustfs/backlog#2873

* fix(decommission): keep a retry rollback's restored record intact

After a retry start rolls back, the restored record is failed, so a
peer worker's own failure mark passed every merge guard and replaced
it. A failure mark now saves through save_current_pool_meta_checked,
which runs the withdrawal check against the record read under the pool
metadata lock and adopts that record instead of writing.

The withdrawal check compared the restored record with the worker's
in-memory last_update, which a rejected save had already raised, so the
peer never adopted the rollback and the watchdog kept resuming it. It
now compares against the running generation (start_time), and a
rejected unresolved-entry save undoes its in-memory change.

save_current_pool_meta_for_decommission_start is also compiled for the
test-util feature again; a test-util helper still calls it, which broke
the workspace clippy and Connect builds.

Refs rustfs/backlog#2873

* fix(decommission): keep an unresolved entry when its own save fails

The rollback of a rejected unresolved-entry save also dropped the entry
on transient failures such as a quorum or lock-fence error, so the
following failure mark persisted a failed record without it and an
operator could clear a decommission that skipped an unreadable object.
Keep the entry and undo only the last_update bump, which is all a
withdrawn worker needs to still see the withdrawal.

Refs rustfs/backlog#2873
2026-10-21 15:45:48 +09:00
唐小鸭 444eb5ddba fix(scanner): admit dirty scope across the bucket-listing window and pace full-walk fallbacks (#8690)
Under sustained writes nearly every cycle fell back to a full walk: scope
admission required the global dirty generation to stay unchanged across the
bucket listing, and after a Complete cycle the dirty-usage wake ran the next
full walk immediately, back to back.

- Track dirty-map removals with a process-local removal generation and admit
  a scoped walk when no entry was removed or replayed during the listing
  (`covers_listed_pending`). Writes to listed buckets in the window join the
  start snapshot; unlisted buckets dirtied after the cutoff stay pending and
  are never cleared by compare-and-remove certification. The strict
  `covers_all_pending` still gates segment-reuse evidence.
- Classify scope fallbacks (removal during listing, generation exhausted,
  dirty-bucket overflow, no listed dirty bucket, baseline unavailable, peer
  scope unavailable), log `scoped_scan_fallback`, and report them in
  usage_freshness as additive serde(default) fields.
- Pace the cycle after a Complete fallback while dirty buckets remain on the
  capped retry schedule clamped to the cycle interval, with the dirty wake
  disabled; the paced timer wake still requests a scope.
- Update the per-bucket convergence design doc.

No persisted format or internode protocol change.

Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-21 13:55:18 +08:00
唐小鸭 bd7b7321a2 fix(replication): read source delete markers on multi-pool stores (#8756)
Delete-marker replication re-reads the source marker by version before sending. Multi-pool get_object_info rejects explicit marker version reads with MethodNotAllowed, so every marker failed verification and was never replicated. Add a defaulted ObjectOperations::get_object_info_for_delete, override it on ECStore with the marker-tolerant snapshot, and route both replication marker reads through it.

Fixes rustfs/rustfs#8746
2026-10-21 13:54:50 +08:00
Chris d122e8e6bc fix(ecstore): ignore child-key prefixes in delete settlement fingerprint (#8751)
* fix(ecstore): ignore child-key prefixes in delete settlement fingerprint

A batch delete of prefix-sharing keys (foo/bar and foo/bar/xyzzy) could
leave foo/bar's commit proof on disk: the child's finalization removed
foo/bar/xyzzy/ between the parent's GC inspection and settlement, the
fingerprint mismatch surfaced as OutdatedXLMeta, and publish_delete_commit
dropped the error. An immediate DeleteBucket then returned BucketNotEmpty.

Fingerprint only this key's protocol evidence (files, UUID dirs and the
transaction dir), retry published cleanup on OutdatedXLMeta with a fresh
inspection, and warn when cleanup is deferred to the recovery hint.

* test(ecstore): cover delete fingerprint narrowing and cleanup retry

Only canonical hyphenated UUID directories count as protocol evidence, so
a child key named as a simple-hex UUID cannot recreate the race. Add
negative fingerprint checks for this key's own changes, and tests for the
published-cleanup retry: success on retry, the attempt bound with the
deferred warning, and no retry after a lost fence.

* fix(ecstore): require canonical UUID names across delete recovery

Discovery, explicit legacy discard and staged-dir checks still accepted
simple, braced and URN UUID spellings. A child key named as the simple
form of a referenced data dir could then be deleted by DiscardLegacy.
Use one canonical-name predicate everywhere and register exactly one
test hook per expected cleanup attempt.
2026-10-21 14:47:34 +09:00
唐小鸭 30c1dffe48 fix(replication): replicate null versions written before versioning (#8757)
The scanner heal gate dropped Some(Uuid::nil()) along with None, so null versions in versioned buckets never reached existing-object replication. Skip only unversioned listings, matching MinIO, and recognize the nil UUID as the null version in the newer-target guard and version comparison.

Fixes rustfs/rustfs#8747
2026-10-21 13:35:47 +08:00
Hauser 2eeaca4cd0 fix(iam): derive bucket location conditions from XML (#8744)
* fix(iam): derive bucket location conditions from XML

Capture the parsed CreateBucket LocationConstraint at the access boundary
and move it into shared authorization conditions independently of signing
and server regions. Cover conditional allow/deny across user, session and
anonymous requests, including absent and spoofed values.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(iam): pass bucket location directly to authorization

Replace temporary request-extension storage with an explicit owned XML
argument to the single authorization core, keeping the public entry point
compatible. Remove the marker allocation and typemap insert/remove path.

Cover Object Lock and Versioning dependent authorizations with an absent
location condition and verify their enabled state through real HTTP.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(connect): isolate bounded object probe storage

Use an independent temporary volume on Linux shared memory when available
so the real endpoint/CLI fixture does not depend on contended CI disk IO.
Await server shutdown before dropping the caller-owned storage directory.
Retain the production cleanup deadline, measurement budget, assertions,
and existing nextest capacity reservation without adding retries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-21 13:35:35 +08:00
Ramakrishna Chilaka 001e92226e test(filemeta,ecstore): cover listing metadata and msgpack decode paths (#8755) 2026-10-21 14:03:23 +09:00
cxymds 65a721d896 fix(ecstore): reindex pool metadata ids after removal (#8733)
* fix(ecstore): reindex pool metadata ids after removal

* ci(nextest): reserve resources for table catalog lock probe

* fix(ecstore): reindex retained decommission references

* fix(ecstore): use typed pool metadata rebuild errors
2026-10-21 13:36:48 +09:00
Hauser f1850f54d4 fix(runtime): isolate configured region per store instance (#8743)
* fix(runtime): isolate configured region per store instance

Resolve the default RegionInterface through the owning ECStore context
and keep ambient fallback only for requests without an explicit context.
Preserve ambient constructors and injected interfaces, with store and
real HTTP multi-instance regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): stabilize quorum and heal crash fixtures

Keep forced-offline disks attached across reconnects and reconnect before every delete retry. Bound SIGKILL fixture I/O while preserving recovery scope, receipts, data integrity, and deadlines; include child execution phases in timeout diagnostics.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(test): clarify forced-offline disk fixture behavior

Describe handle retention instead of claiming a newly reconnected health
tracker inherits the test outage. Execution and assertions are unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-21 13:06:54 +09:00
cxymds 2c55d9c8f0 fix(ecstore): allow repeated marker deletes at write quorum (#8750) 2026-10-21 11:31:59 +08:00
Chris 5efa6c2173 fix(rebalance): move tier free versions with their entry (#8753)
* fix(rebalance): move tier free versions with their entry

Rebalance built its version snapshot with MetaCacheEntry::file_info_versions,
which drops tier free versions, while the source-cleanup preflight reads the
exact snapshot that keeps them inline. Any entry holding a free version (left
by deleting a transitioned version) therefore failed the preflight on every
pass and was deferred forever, so the source pool could never be drained of it.

Mirror decommission: keep free versions in the snapshot, skip lifecycle and
the last-delete-marker policy for them, and move them to the target as tier
metadata through decommission_tiered_object. A free version reads as a delete
marker with no transition status, so the migrator routes it explicitly;
written as a delete marker it would drop the remote owner and orphan the
remote object. A moved free version may be consumed from the source before
cleanup, like decommission; free versions hold no pool data and stay out of
the rebalance stats.

Refs rustfs/backlog#2873

* test(rebalance): cover tier free version movement edge cases

Pin the lifecycle skip, the delete-marker count, the allowed-missing and
not-found consumption paths, the stats exclusion, multiple free versions per
entry, and the unversioned superseded source's free version moving on the
next pass.

Refs rustfs/backlog#2873
2026-10-21 12:25:30 +09:00
Hauser 2110348e58 fix(metrics): expose allocator observation availability (#8748)
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-21 10:34:27 +08:00
cxymds 2253ba9009 fix(tier): restore private endpoints and forced config removal (#8725)
* fix(tier): restore private endpoints and forced config removal

* test(tier): align proof failure with force semantics
2026-10-21 10:21:27 +08:00
GatewayJ 72d5c5bf2f fix(table-catalog): support manifest-only rewrites (#8731)
* fix(table-catalog): support manifest-only rewrites

* fix(table-catalog): preserve manifest rewrite metadata
2026-10-21 09:29:23 +08:00
Hauser 13c37bf8b9 fix(s3): validate bucket location constraints (#8739)
* fix(s3): validate bucket location constraints

Reject explicit bucket locations outside the configured server region
before mutation admission, preserve site-default and EU alias compatibility,
and return an empty GetBucketLocation element for us-east-1.

Cover matching and rejected locations over real S3 HTTP requests, including
rejection without creating a bucket.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): refresh bucket region profile selection digests

Include both new region regression tests in the smoke and Darwin/Linux
full selection pins without changing filters or checker behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-21 03:28:23 +09:00
Hauser 38595c1909 fix(metrics): preserve S3 operation across detached tasks (#8738)
Establish the first bucket operation before fresh-task dispatch and carry its bounded value into nested config and background work. Preserve cancellation, permit ownership, lock order, and error mapping without importing diagnostic owner or sampling instrumentation.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-21 01:22:28 +09:00
cxymds 9b41835580 fix(decommission): allow retry after unresolved failure (#8737) 2026-10-20 23:07:06 +08:00
cxymds a5ae294f42 fix(quota): refresh peer metadata after admin quota changes (#8735)
* fix(quota): refresh peer metadata after admin quota changes

* test(e2e): update distributed selection for quota regression
2026-10-20 23:06:18 +08:00
GatewayJ 4dba635d48 fix(table-catalog): authorize warehouse location claims (#8736) 2026-10-20 22:05:29 +08:00
Chris 7cc4a9dbd0 fix(ecstore): leave no source free version for a moved tier version (#8732)
* fix(ecstore): leave no source free version for a moved tier version

Data-movement source cleanup deletes each source version separately and
never set skip_tier_free_version, so every transitioned version that
decommission or rebalance had already copied to a target pool left a new
tier free version on the source. Decommission then failed its post-check
for any pool with tiered objects; rebalance left the record behind. The
record claims the remote object the target copy still reads and stays
blocked behind that live reference.

Decommission and rebalance now pass the identities of transitioned
versions whose copy committed on a target, and the cleanup skips the
free version for exactly those. A transitioned version that did not
move, such as one lifecycle expired during the move, still gets its free
version because that record is its only remote-delete owner.

Refs rustfs/backlog#2873, rustfs/backlog#2872

* fix(ecstore): keep the free version of a superseded moved source

Data movement treats a source null version as complete when a newer
unversioned object already sits on the target, without copying it.
That outcome returned the same Ok as a committed copy, so a restored
transitioned null version superseded this way was recorded as
transferred and lost its source free version, leaving its remote object
with no owner.

The copy paths now return DataMovementCopy::Committed or ::Superseded,
and decommission and rebalance record only committed copies of
transitioned versions. A superseded source keeps its free version.

Tests cover the superseded restored null version for rebalance and
decommission, a restored and a null transitioned version moving by
rebalance, an entry with one lifecycle-expired and one moved version,
the per-version skip with two versions of one key, and a directory
object. The tiered stale-target routine test from #8724 now expects the
decommission to complete. The ILM ownership contract and the
decommission notes describe the moved, superseded, expired and legacy
cases.

Refs rustfs/backlog#2873, rustfs/backlog#2872

* test(ecstore): build the decommission ILM fixtures without s3s

The superseded and expiry-mix decommission tests named s3s request and
lifecycle types directly, which added pools_test.rs to the ecstore
files that reference s3s and failed the footprint ratchet. Set the
restore request through ObjectOptions, and store the lifecycle rule as
bucket metadata XML and load it with get_expiry_configs, as the
decommission routine does.

* test(ecstore): pin restored decommission copies and multipart supersession

Decommission recorded a committed data copy of a restored transitioned
version without any test failing if the record were dropped; add a
routine-level decommission of a restored version that checks the post-
check passes, the source keeps nothing, and the target keeps the full
remote tuple. Add the multipart variant of the superseded restored null
version (rebalance), and a unit test that separates committed,
superseded and mismatched targets in the overwrite resume adjudication.
2026-10-20 22:20:12 +09:00
Hauser 928a369445 fix(obs): skip address discovery for initialized node identity (#8730)
* fix(obs): skip address discovery for initialized node identity

Check the nonblocking node name before resolving the local IP, retaining the final name recheck after fallback. Add portable publication regressions and Linux worker-local netlink fault coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(obs): keep identity contention checks in common

Exercise the real held-writer getter inside its owning module. Publish through the existing runtime helper from obs tests, retaining the kernel netlink failure and recovery coverage without bypassing architecture guards.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(obs): scope unsafe allowance to seccomp test helper

Document the bounded prctl FFI safety contract and allow unsafe code only
inside the Linux fault-filter installer so workspace unsafe-code denial
continues to apply everywhere else.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-20 20:17:55 +09:00
Chris 5313b72baa fix(decommission): run the leader-side start to completion (#8723)
* fix(decommission): run the leader-side start to completion

A forwarded decommission start awaits `ECStore::decommission` inline in
the RPC handler. When the forwarding node's RPC times out, the stream is
reset and the handler future is dropped wherever it is. Dropped after
`pool.bin` records the pool as active but before the peers reload and
the worker is spawned, the start left the pool active with no worker and
peers on the old metadata, and a retry was rejected as already active.
An admin client that disconnects can do the same.

Run the start in its own task and let the caller await only its result.
The start then always reaches its end: it fails before persisting
anything, or persists the pool, has the peers reload and spawns the
local workers, rolling back if the reload fails. A retry still cannot
start twice: it waits on the start gate held by the running start and
either reports the busy gate or is rejected because the pool is active.

`decommission` now takes `self: &Arc<Self>` and spawns workers on that
store instead of looking up the global object store, which is the same
store in production and makes the path testable; the unused
`require_decommission_store` helper is removed.

Refs rustfs/backlog#2873

* fix(decommission): detach the leader-side start from its caller

9ac4c10de carried only the reproduction (the testable entry point, the
peer-reload pause point and the proof test). This adds the fix it
describes: `ECStore::decommission` spawns the start as its own task and
awaits only its result, so a caller that stops waiting, such as a
forwarded start whose internode RPC timed out, can no longer drop the
start between persisting the pool as active and having the peers reload
and the local worker start.

Also prove a retry while the first start runs cannot start the pool
twice, update the operator and architecture docs for the new behavior,
and shrink the error(format!) ratchet baseline for the removed helper.

Refs rustfs/backlog#2873

* fix(decommission): single-flight start, recheck pool state under gate

Review of 3a5276c5b:

- A start re-checks the pool state under the start gate before it
  cancels routines, so a start racing another one can no longer cancel
  the worker the other start just spawned.
- Only one start runs per instance at a time: another request, including
  the retry of an operator whose request timed out, is rejected as
  already running instead of stacking a second detached start and its
  preflight heals. The start runs through `run_detached_mutation`, and a
  failure nobody waits for any more is logged.
- Tests: the proof test pins its drop after the save and requires a
  worker that persists a cancellation; the retry test really drops the
  first caller; new tests cover the under-gate recheck and a peer reload
  failure after the caller is gone. That last one shows the start
  rollback cannot clear a fresh start because the pool metadata
  stale-write guard refuses it; the gap is asserted and documented and
  tracked in rustfs/backlog#2873.
- The start tests run on their own large-stack thread: the io_uring lane
  (`uring_`, which also selects `during_`) runs them with libtest's 2 MiB
  default stack, where the proof test overflowed.
- Docs: single-flight starts, status and cancel during preflight, the
  rollback gap and its recovery, and the release requirement.

Refs rustfs/backlog#2873

* fix(decommission): turn away local start retries at the start slot

Second review of 29253430a:

- The leader-local admin pre-check refreshed rebalance metadata under
  the start gate before the start slot was checked, so a local retry
  waited out its 20 s gate budget behind a running start (which holds
  the gate through its peer reload) and then got a 503. The pre-check
  now checks the slot first and returns `DecommissionAlreadyRunning`,
  which the handler reports as a 400 that keeps the reason.
- Prove the reload-failure path really ran in the detached start (the
  injection is consumed), add a test that a fresh start's failed
  rollback reports both the reload error and the stale-write refusal,
  and assert an abandoned start stays tracked for shutdown.

Refs rustfs/backlog#2873
2026-10-20 18:20:13 +09:00
Chris f98d268f84 fix(decommission): retry every stale target selection without failing (#8724)
* fix(decommission): retry every stale target selection without failing

The tier free-version copy path counted a version as failed as soon as
a capacity failure came back and then handed the error to the entry
retry, so a retry that succeeded left the decommission failed. Return
the failure without counting it, like the other copy paths.

A selected pool can also stop fitting the mutation before admission
without going over a budget: the reservation estimate is exhausted, the
pending intent no longer fits the committed-copy budget, or a temporary
write no longer fits the headroom. Report these as "no longer fits the
selected mutation" and treat them like an over-budget pool under one
StaleTargetSelection class: retry, select again, defer once the retry
budget is spent.

Also refresh the over-budget gauge of every allocation before rejecting
growth, correct the sticky-target doc on stranded markers, and cover
each copy path, the pinned path and both retry branches with tests.

Refs rustfs/backlog#2873

* fix(decommission): name the last deferral cause in the pause reason

A round that ends with deferred entries pauses the routine with a
generic "deferred entries remaining" reason, so a pinned entry that no
longer fits its pool hides its cause in the logs. The pause reason now
appends the last deferral's error.

Keep the committed-copy check on the pending intent fatal: growth and
owner admission reject any mutation that would fail it first, so it
only fires on an inconsistent ledger.

Tests:
- a multipart part that does not fit target 1's temporary-copy
  headroom is deferred and pauses with the headroom shape;
- a pinned object whose remaining allocation is below its peak pauses
  with the exhausted-estimate cause, and the over-budget pinned test
  checks its own cause;
- the stale-target tests assert where the pause lands: the held entry
  is on neither target, the anchor is already on target 1, and target
  1 holds exact inflight bytes in the multipart cases;
- share the stale-target store, barrier and pinned-copy setup.

* fix(decommission): prefer a stale selection as the pause cause

The pause reason named whichever deferral of the set finished last, so
a transient busy gate could hide another entry's permanent shortfall.
A stale target selection, which waiting does not clear, now outranks
transient contention, and the reason reads "; deferral cause: <error>".

Tests:
- a pinned object that no longer fits and a later object whose gate is
  busy, in the same set, pause with the pinned cause; a unit test checks
  both orders;
- the tiered and intent-conflict tests check where the pause lands, and
  the tiered test asserts the known post-check failure so that fixing
  the source cleanup defect flips it.
2026-10-20 17:40:16 +09:00
Ramakrishna Chilaka f4f1743d04 perf(ecstore,filemeta): avoid inline-data reads and redundant decodes in list_objects (#8706) 2026-10-20 16:33:02 +09:00
Chris 8acfde3842 test(quota): cover early-reload routing and snapshot age boundaries (#8722) 2026-10-20 14:05:55 +09:00
overtrue f320f791a4 test(decommission): prove the shared gate budget and each bounded wait
Second review of 652c56875..0ef02a485:

- The operator doc and the budget comment claimed a forwarded start that
  outlives the internode RPC timeout still completes on the leader. The
  leader drops it wherever it is, possibly after pool.bin records the
  pool as active but before its worker starts or the peers reload.
  Document that state, the watchdog that resumes the worker, and the
  checks to run; the cancel-safety fix is tracked in rustfs/backlog#2873.
- Prove the budget is shared across waits (paused clock: a 60 ms wait
  leaves 40 ms for the next gate).
- Prove the rebalance-refresh start gate and activation gate waits and
  the decommission() entry are bounded and named.

Refs rustfs/backlog#2873
2026-10-20 13:29:14 +09:00
overtrue 4d2e362e23 fix(decommission): count only gate waiting in the start budget
Review of 652c56875 found that the 30 s start budget was a wall-clock
deadline fixed when the start began, so slow preflight work (bucket heals,
capacity scans) could spend it before the activation gates were reached.
A gate held for a few milliseconds then failed the start at once, and a
retry repeated the preflight.

- Replace the deadline with a budget that only counts time spent actually
  waiting for a gate; a free gate is always taken, and the lock timeout
  reports the time waited.
- Lower the budget to 20 s, below the default 30 s internode RPC timeout,
  so a forwarded start gets the leader's answer instead of an RPC timeout.
- Name each bounded wait (rebalance metadata refresh or decommission
  activation, and which gate), and route every start-gate rebalance load
  through one load_rebalance_meta_within.
- Move the admin pre-check error mapping into a function and test it.
- Add proof tests: a slow preflight followed by a brief gate wait still
  starts; the activation wait and the admin pre-check are bounded and
  named; the rebalance status read does not wait for an in-flight
  rebalance entry.
- Correct the operator doc: preflight bucket heals are outside the budget,
  and forwarded starts report a generic internal error.

Refs rustfs/backlog#2873
2026-10-20 13:29:14 +09:00
overtrue a55f2f671b fix(decommission): keep status reads and starts off the movement gate
The data-movement operation gate is a write-preferring RwLock. A scanner
publication lease stores a read guard on the leasing node, and the
coordinator's lease-scoped rename, delete and validation on that node
queued a second read. A movement writer queued in between (a decommission
start, a status refresh, a peer reload) then waited for the lease while
the coordinator's second read waited for the writer, so both stalled
until the 60 s lease TTL. Admin status reads also took the movement
writer, so every status call waited for the slowest in-flight copy or
cleanup and stalled new movement reads while it queued.

- Lease-scoped target admission shares the read admission the lease
  already holds instead of queueing a new read.
- Pool and rebalance status reads publish under the gates only when they
  are free; otherwise they merge the persisted metadata into a private
  copy and publish nothing.
- A decommission start bounds its start, activation and movement gate
  waits with one 30 s budget and fails with a retryable lock timeout
  before persisting anything; the admin handler maps it to 503 with
  Retry-After.

Refs rustfs/backlog#2873
2026-10-20 13:29:14 +09:00
Chris e7e53f07b2 fix(decommission): scope temporary-copy budget checks to the growth pool (#8721) 2026-10-20 13:28:51 +09:00
Chris da7fde0e62 fix(quota): stop pinning quota e2e readiness to the scanner leader (#8719) 2026-10-20 13:28:22 +09:00
Chris 6127deaa83 test(obs): pin daily rotation boundary test to fixed timestamps (#8716)
The test derived its inputs from Zoned::now() and assumed that now - 60s is still on the same local calendar day, so it failed deterministically in the first minute after local midnight. Build the inputs from fixed local wall-clock times instead, covering same day, previous day and the 23:59:59 -> 00:00:00 crossing.
2026-10-20 11:52:26 +09:00
Chris ea596b2e2d fix(decommission): adopt target intents left by a killed leader (#8710)
* fix(decommission): adopt target intents left by a killed leader

A decommission leader killed after persisting a target capacity intent and
before finalizing it left that intent in the exclusive pending slot of the
still-active reservation. The resumed routine derives the same reservation
identity, but every other mutation on that target was rejected as "another
mutation" until the interrupted object happened to be retried on the same
target, so the resumed decommission exhausted its retries and stalled.

At routine start, before any worker runs, the resumed routine now adopts the
reservation's pending intents: under the target fence (or the legacy pool
metadata fence) it moves each one into a mutation-scoped inflight marker that
keeps its identity and bytes. The interrupted mutation's retry settles the
marker through the existing commit, temporary-release and equivalent-target
paths, and an equivalent committed target now settles a mutation's own marker
when no pending intent remains. Adoption is pinned to the persisted
decommission start time and reservation identity, never runs mid-incarnation,
and a failure pauses the routine instead of marking the decommission failed.

* fix(decommission): keep adopted intents within budget and settle them once

Address the adversarial review of the leftover intent adoption.

- Merge an adopted intent into its mutation's staged marker by summing only
  while the target's temporary-copy budget allows, never below the larger of
  the two. A multipart completion moves its staged parts into the object, so
  summing them could exceed the budget and pause every later routine start;
  the adopted marker now exceeds the budget only where the completion's own
  finalize would have. Reuse resolve and record_inflight for the move.
- Distinguish a temporary release that proved the exact committed target from
  one that could not prove the target identity. When the committed mutation no
  longer has a pending intent, release only the bytes the cleanup freed and
  keep the adopted completion for the equivalent-target reconciliation.
- Let an equivalent target settle a mutation's own marker while another
  mutation holds the pending slot, instead of failing on that unrelated intent.
- Cover real leader kills before and after the target commit at routine level
  for both lock models, an injected adoption failure that pauses and then
  completes, an adopted multipart completion surviving a background abort, and
  a live legacy intent. Routine tests reuse the large-stack test helper.

* fix(decommission): pause instead of failing on target slot contention

The S00 leader-kill run with the null-version fix merged ended the resumed
decommission as failed. A re-run of a versioned key reconciled its delete
marker against an already charged target copy while another worker of the
same incarnation held the target's pending slot between a failed write and
its retry. The reconciliation rejected that unrelated intent, the entry
context wrapper flattened the error to a string the intent-conflict retry
could not recognize, and the bucket runner reported a sibling bucket's
OperationCanceled instead of the capacity block, so the routine marked the
decommission failed.

- An equivalent-target reconciliation, including the temporary checkpoint
  variant, no longer depends on another mutation's pending intent: a target
  holds one pending intent at a time, so this mutation has none to settle.
- The entry context keeps the typed cause, and the capacity-block and
  intent-conflict classifiers also recognize the rendered message that an
  error clone keeps.
- Concurrent bucket and listing failures keep the root cause over sibling
  cancellations, so a capacity block pauses the routine.

* test(decommission): delete marker copy waits for a busy target gate

The Linux run of the S00 matrix showed the delete marker copy meeting a busy
target capacity gate without any leader kill. The copy already waits on that
gate like an ordinary object copy; pin it with an entry-level test that holds
the gate, checks the entry keeps waiting, and expects the key to migrate once
the gate is released. Share the versioned delete marker fixture with the
foreign pending slot test.

* fix(decommission): keep an adopted completion unless the target is absent

Close the remaining path where an adopted completion could lose its record
before being charged. A multipart cleanup that found a target it could not
prove as the exact owned copy still dropped the whole marker of a mutation
without a pending intent, after which the equivalent-target reconciliation
would treat the commit as already charged.

Any cleanup that did not prove the exact target absent now releases only the
bytes it freed when the mutation has no pending intent, and leaves the rest
for the reconciliation. This makes the three-state release unnecessary, so
the release proof goes back to the existing clear-pending flag.

* chore(ci): shrink the error format ratchet baseline for pools.rs

9eae47ce8 removed one Error::other(format!) call site from crates/ecstore/src/core/pools.rs, so the ratchet requires the baseline to drop from 178 to 177.

* fix(decommission): keep adopted completions in their own capacity marker

An adopted completion shared the mutation's staged marker, so whether
it survived a cleanup depended on the pool-wide free-space delta, and a
cleanup that freed little space left staged bytes behind and repeated
forever.

Record each adopted intent under a mutation id derived from the
interrupted mutation (decommission_adopted_intent_mutation_id) as a
separate temporaryMutations entry. A commit or an equivalent-target
reconciliation settles both markers, a temporary cleanup releases the
staged marker as before, and only a cleanup that proved the exact
target absent drops the adopted one. While the target holds only that
adopted completion, a cleanup keeps the delayed-release exemption of
the published pending intent it replaced. Older nodes see an ordinary
temporary mutation they never release, which only over-reserves.

Also log capacity_intents_adopted as soon as the adopted ledger is
durable, merge the root-cause helpers into decommission_root_cause,
drop the decommission_capacity_blocked: prefix fallback, and log the
already-charged equivalent-target branch at debug level.

Refs rustfs/backlog#2873, rustfs/backlog#2872

* fix(decommission): return uncommitted retries to their adopted target

A retry that had not committed yet went to the reserved target with the
most remaining allocation. When that was not the target holding the
mutation's adopted completion, the commit settled only its own target
and the adopted marker stayed inflight until the reservation was
released.

Prefer a reserved target that holds the mutation's adopted marker while
it still fits the object (decommission_reserved_target_choice).

Also cover the remaining settlement paths of the derived marker
(non-growing replacement, equivalent temporary reconciliation and its
discovery gate, scanner backlog retirement), repeated adoption of one
mutation and the zero-byte staging record in both lock models, drop two
conditions that always equal another one, and correct the adopted-id
doc comment about nodes without adoption.

Refs rustfs/backlog#2873, rustfs/backlog#2872

* fix(decommission): count a completion adopted twice only once

A mutation killed twice leaves an adopted completion and then a new
intent for the same object version. Adoption added the two, so an
11-byte object counted 22 inflight bytes and could trip the capacity
check once later commits shrank the budget. Both reserve the same
completion, so merge them by their maximum before adding staged parts;
a single adoption is unchanged.

Also assert in the three-pool sticky test that no entry failed, the
retry is charged on its adopted target and the source is cleaned, run
it for the legacy lock model too, stop saying operators rely on the
adoption log before rollback, and document that the selection fallback
cannot help an adopted completion.

Refs rustfs/backlog#2873, rustfs/backlog#2872

* fix(decommission): log where an adopted retry is placed

An e2e run could not show whether a resumed retry went back to the
target holding its adopted completion. Target selection now logs
adopted_target_reused when it does, and adopted_target_bypassed with
the adopted target pools when the object no longer fits there and the
adopted marker stays behind. Both carry the mutation id and the source
and target pools.

Refs rustfs/backlog#2873, rustfs/backlog#2872
2026-10-20 11:28:22 +09:00
Chris 10a9b158d7 fix(ecstore): address the null version exactly during data movement (#8711)
* test(ecstore): pin null-version migration behind a delete marker

Decommission and rebalance must move a pre-versioning null version that
sits behind a delete marker, with its content and SHA-256 intact, before
removing the source entry (rustfs/backlog#2873).

* fix(ecstore): address the null version exactly during data movement

Decommission and rebalance read each source version with
`version.version_id.map(to_string)`. A null version, written before
versioning was enabled, has no version id, so the read used `None`,
which selects the latest version. Behind a delete marker that read
returned not found, the mover counted the version as already gone, and
source cleanup deleted the still-present null version with the entry.

Address the null version by the nil UUID, the existing exact selector,
on every data-movement lookup: the source read, the target put, new
multipart and complete options, the target resume lookup, and the
source revalidation before publication. The stale-target replacement
check treats the nil selector and an id-less null version as the same
identity, so unversioned buckets keep their current behavior. Target
pool selection still places a null version with the object's latest
version, as before, so a later null-slot overwrite replaces it there.

A version the mover skips as not found must also really be gone before
cleanup. It is dropped from the expected cleanup snapshot, so the
preflight reports SourceChanged while that exact version still exists
and the existing retry and exhaustion path retains the source.

Refs rustfs/backlog#2873, rustfs/backlog#2872

* fix(ecstore): prove multipart null-version staging absence exactly

The multipart staging cleanup proof in
abort_multipart_uploads_for_data_movement looked up the target with the
abort options' version id. Those options keep an id-less null version
as `None`, because the staged upload identity is derived from them, so
the proof resolved the target's latest version. Versions move newest
first, so it found the delete marker or newer version already moved and
refused every attempt with "cannot prove the exact staged identity is
absent". The legacy capacity model runs this proof on every attempt and
the exact model after any interrupted upload, so a multipart null
version behind a newer version could never be decommissioned. Look up
the exact null version for the proof and keep the abort options as they
are.

Cover the paths review found unprotected: multipart null versions under
the legacy model and after an interrupted upload, null versions behind
a newer version (lost without any not-found), the cleanup guard for
versions reported as not found while they still exist (decommission
read and delete-marker copy, rebalance read, and a control where the
version really vanished), and a retry after the null version committed
but before cleanup. The decommission fault hook can now inject a chosen
error, and rebalance gains a test-only source read fault.

Refs rustfs/backlog#2873, rustfs/backlog#2872
2026-10-20 10:12:59 +09:00
Chris 6f582b0548 fix(heal): decide object absence from storage evidence, not source (#8697)
* test(heal): add failing regressions for a source-independent absent verdict

Add the regression tests for backlog#2900 that fail on the current code,
where the same object-level not-found is treated as deleted on the plain
heal path but as recreate_failed on the recreate path, and where the
erasure-set healer, the task predicates and the error helpers disagree on
wrapped not-found and transient inputs.

Also add the per-object disposition cache module with its unit tests, the
manager admission tests for it, the compat register entry for the display
boundary, and the admission-invariant notes. The module and the manager
tests are not declared yet, so nothing is wired into the build.

* fix(heal): decide object absence from storage evidence, not request source

Replace the five heal error classifiers (erasure-set healer outcome
mapping, task-level not-found and lock predicates, error.rs recovery
helpers and the storage existence-check matcher) with one typed
classify_heal_error. An object-level not-found from the locked storage
read is Absent for every source and every recreate_missing value; the
recreate path, the usage-observation and scanner object-dir whitelists
are gone, and display-text matching survives only in the marked
classify_error_display boundary.

Add a bounded, expiring per-object disposition memory consulted before
queue admission so a best-effort producer cannot requeue an absent or
terminally failed object on every tick; durable MRF, admin, forced,
high-priority and dry-run requests bypass it. The read-repair producer no
longer forces recreate_missing.

* fix(heal): keep rendered grace windows out of task retries

A grace window rendered into a task message belongs to an attempt that
already finished; its owner schedules the next one. Only typed grace
errors replay the task, as before the classifier merge. Update the
ecstore read-repair black-box test: the producer no longer forces
recreate_missing.

* fix(heal): classify ambiguous legacy delete residue as permanent explicitly

* fix(heal): scope disposition memory and bound absence parsing
2026-10-20 00:11:49 +08:00
Chris f063a8e862 fix(readiness): hold startup readiness until peers re-admit the node (#8704)
A restarted node published FullReady from its own view while survivors still counted its drives unavailable (stale DNS for the old pod IP, then the Offline->Returning->Online hysteresis). A rolling update could then stop the next pod and cost a set its read quorum (#8669).

Add a read-only PeerDriveAdmission internode RPC that reports how a peer sees another host's drives, and gate the startup FullReady transition until a strict majority of each erasure set's peers count this node's drives online. The gate shares the startup budget, releases after 15s of unreachable peers, skips immediately when older peers cannot answer, and can be disabled with RUSTFS_STARTUP_PEER_READMISSION_ENABLE=false.
2026-10-20 00:28:48 +09:00
Chris 83fb90cb6e ci: move API-only and summary jobs to hosted runners (#8707)
Apply the placement rule in .github/workflows/README.md to the 22 jobs
still on sm-standard-2 that only call APIs, read workflow metadata, print
summaries or scan published images, including cla.yml, the busiest
workflow. Moving the ODM cloud-source lane also stops it restoring the
ci-odm-interop key across images, since its writer already runs on
ubuntu-latest; targets-live gets a -hosted cache key for the same reason.
2026-10-20 00:28:24 +09:00
Chris 473cddd7dc ci: gate pull requests on Clippy and warm its check artifacts (#8708)
Clippy ran only after merge, so lint regressions merged freely and then
failed every main run until fixed (10 of 12 main runs on 2026-10-17/18).
Run the workspace Clippy step on every event, warm its check-mode
artifacts in cache-warm with the identical invocation so pull requests
re-check only what they touch, and pin the unconditional step in the
ci_gate contract and its self-test.
2026-10-20 00:27:40 +09:00
Chris 9a1f3d2c4c fix(quota): keep ledger-admitted writes independent of scanner usage (#8699)
Hard quota admission is owned by the durable reservation ledger (decision
D3, rustfs/backlog#2872), but the request-path quota check still read
scanner usage on every write and mapped `UsageUnavailable` to a retryable
503 even for buckets the ledger admits. For a durable quota the checker now
treats scanner usage as reporting input: an unavailable lookup yields
`current_usage = None` and `allowed = true`, counted and logged at debug
level, and admin quota stats report the usage as null instead of zero.
Buckets without a reservation protocol keep the fail-closed snapshot path.

Quotas saved before the reservation protocols are re-saved under protocol
v2 with the same limit, type and created_at. The write goes through a new
guarded save that keeps the admin save's lifecycle/transaction guards,
incarnation check and fleet capability proof and additionally refuses when
the quota revision moved since the cached read, so a concurrent quota edit
wins. It is triggered lazily from the quota read path (background, one
attempt per bucket with a retry interval, never blocking the read) and in
bulk by `POST /rustfs/admin/v3/quota-migration`, which reports every bucket.

The ADR's Consequences section inventories the remaining scanner-usage
"authority" consumers with their meaning after D3, retirement condition and
compatibility window; the operations guide documents the migration and the
report-only usage semantics.

Refs rustfs/backlog#2904, rustfs/backlog#2872, rustfs/rustfs#8678,
rustfs/rustfs#5716, rustfs/rustfs#7350

Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-19 22:45:08 +09:00
Chris 7c75f319bc fix(scanner): stop treating an ordinary PUT as data movement in cycle termination (#8696)
* fix(scanner): stop treating an ordinary PUT as data movement in cycle termination

A continuously written deployment never reached a `Complete` scanner cycle
(#8494, #8305, #7108). Three places in the termination path vetoed the cycle
for an ordinary PUT; each now lets the write shade its bucket, the path
per-bucket certification (#8542) built, and keeps the data-movement fences.

1. Usage publication is fenced by data movement only.
   `scanner_data_usage_publication_blocked` no longer includes
   `namespace_commits_pending()`, so the post-walk probe, the local defer
   decision, the post-lease recheck and the usage-store route probes stop
   superseding or deferring a cycle for a pending rename fanout. The scanner
   publication leases (acquire, validate, target guard) drop the
   pending-commit rejection and the namespace-generation staleness check and
   keep the movement generation, the operation gate, generation exhaustion
   and the purpose/target restriction. A test-util probe lets fixtures wait
   for tails directly.

2. The plan identity is split into a coverage identity and a reuse identity.
   `scanner_activity_coverage_digest` (peer set, protocol, topology, movement,
   and a remote peer's generations) is what a bucket checkpoint is started
   and resumed under, through `ScannerBucketScanPlan::checkpoint_coverage_digest`,
   `DataUsageCacheReuseOptions::checkpoint_plan_digest` and
   `prepare_bucket_checkpoint_with_coverage`. The reuse identity
   (`scanner_activity_structural_digest`) keeps `instance_id` and
   `maintenance_generation`, so a restart or a maintenance change still
   invalidates the baseline and the set caches; a completed sweep is stored
   under it. `publication_blocked` leaves every digest and stays a guardrail
   veto. A remote worker still resumes under the single digest on the wire.

3. A bucket walk that stops early no longer makes the cycle `Incomplete`
   when the bucket has a certified value. The set scan stands in with the
   bucket's flattened value from its previous complete set cache
   (`carry_partial_bucket_values`), the cycle reports it pending
   (`ScannerBucketScanStatus::PartialCarried`), keeps it in the dirty map,
   and the next scoped cycle walks it because the baseline's pending list
   joins the dirty buckets. A bucket with no certified value keeps the cycle
   `Incomplete`; failed buckets are unchanged.

No persisted file, generation, receipt or environment variable is added.
Refs #8494, #8305, #7108, rustfs/backlog#2903.

* chore(ecstore): gate the namespace commit probe to test builds

The scanner publication barrier no longer consults
InstanceContext::namespace_commits_pending, so it has no production
caller left; keep it for tests and the test-util facade only.

* test(rustfs): observe namespace commit ownership directly in node_service tests

The scanner usage publication barrier is movement-only now, so it no
longer reports a pending rename. The node_service tests that assert a
cancelled handler keeps its physical owner, or that wait for rename
tails to drain, read the namespace commit state through a test-only
ECStore probe instead.
2026-10-19 22:23:59 +09:00
Chris c48ff0bbbd test(connect): hash test provenance before listener peer-cred probe (#8703) 2026-10-19 19:50:44 +09:00
Hauser 05b483b31a fix(storage): preserve scanner checkpoints and bound MRF journal I/O (#8692)
* fix(scanner): preserve checkpoints on identity read failures

Distinguish confirmed missing incarnation sidecars from lookup errors before
cache preparation. Preserve durable checkpoint progress when identity reads
are unavailable and report post-scan uncertainty separately from replacement.

Reuse the transaction-fenced reader for optional and required lookups while
retaining existing missing, malformed, system-bucket and lock-loss behavior.
Verify preservation and recovery through a real coordinator with the identity
sidecar write-locked, including unchanged cache revisions and dirty work.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): share required incarnation conversion

Keep the missing-incarnation error contract in one conversion used by both
required lookup entry points. This removes a duplicate formatted error site
and satisfies the existing error-format ratchet without changing its baseline.

Pin the full missing-identity error text in the real SetDisks regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): bound cross-disk MRF checkpoint I/O

Run independent journal disks with bounded concurrency while retaining each
replica's payload, companion and manifest ordering and the mirror phases.
Share encoded bodies, deduplicate local roots and collect every result
without changing receipt deadlines or persistence failure semantics.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-19 18:16:49 +08:00
Chris f3cef98afc fix(scanner): anchor the bitrot cadence and budget per-disk I/O (#8701)
* test(scanner): pin Normal mode when no bitrot start is recorded

Regression for rustfs/backlog#2905: three consecutive cycles that find no
recorded bitrot start currently run Deep through the selection window, and a
Normal cycle never records the cadence start.

* fix(scanner): anchor the bitrot cadence and budget per-disk I/O

Decide the scan mode from a recorded Deep window start only. A cycle that
finds no recorded start runs Normal and anchors the bitrot cadence in
process memory instead of opening the selection window; the Deep cycle that
follows the elapsed cadence records the window start exactly as before. A
missing record used to be read as a window that started at cycle 0, which
ran Deep for deep_window_cycles cycles whenever the start could not be read
back (rustfs/rustfs#8015). The recorded start is also remembered across a
skipped or failed save, and /v3/scanner/status reports the cadence state
under a new bitrot object.

Add a per-local-disk background I/O budget (operations and bytes per second,
defaults by detected media, auto|off|<rate>) that DynamicSleeper charges
before directory listings and object metadata reads as a hard ceiling
layered on the foreground-priority disk admission gate, with runtime config
keys, per-disk status, cycle metrics, pacing pressure, usage_freshness,
Prometheus series, Helm values, admin config help, and operator docs.

Refs rustfs/backlog#2905, rustfs/rustfs#8015, rustfs/rustfs#8377.

* test(scanner): expect Normal retries on erasure sets without a bitrot start

The pending-PUT retry fixture asserted that a 4-drive set revisits every
bucket because the cycle ran Deep. Without a recorded Deep window the cycle
now runs Normal on every layout and reuses the cold inventory.
2026-10-19 18:15:03 +08:00
Chris 2b57fa8758 fix(heal): block replacement and root records per entry (#8700)
Count only post-format rebuild failures against the replacement retry
budget; handoffs, format fences, outages, grace waits and conflicts keep
their durable reason without spending an attempt. Retired generations
(committed handoff or approved legacy migration) no longer own their set,
so fresh admission for other slots proceeds, and a committed predecessor
becomes collectible once its successor is verified.

Block discovery conflicts per generation instead of per set, re-evaluate
them on every pass, and expose the blocked entries with reasons. Report
root heal inventory conflicts per record and keep replaying the others.
Revalidate blocked checkpoints hourly with the current reader and clear
markers the reader accepts. Return a dedicated error for ambiguous legacy
delete residue instead of the retryable SlowDown.

Implements rustfs/backlog#2902. Fixes rustfs/rustfs#8664. Addresses
rustfs/rustfs#8366, rustfs/rustfs#8565, rustfs/rustfs#8645 and part of
rustfs/rustfs#8608.

Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-19 18:12:43 +08:00
Chris 20d9277ec3 ci: return pull-request build jobs to 2 after an IO-bound timeout at 3 (#8698)
The first pull-request run at CARGO_BUILD_JOBS=3 that met a contended
node (39193827281) hit the 75-minute nextest timeout before any test
started. Its sampler shows cargo spinning for 70 minutes with the pod's
io.pressure "full" at 86-88% and cpu.pressure at zero, with node-level
IO pressure already at 36% when the step began: three concurrent
workspace-scale compile units saturate the pod's shared storage, the
mechanism the original #5394 mitigation named. Keep 3 on main pushes and
dispatches, where three runs succeeded, and 2 on pull requests, in both
the nextest and the serial ILM step; pin the ILM budget guard back to
that expression and record the evidence in the step comment.
2026-10-19 17:54:57 +08:00
Chris 2534b6175b fix(scanner): startup gates back off, rebuild and run degraded (#8695)
* fix(scanner): give startup gates automatic exits instead of a pause

Every fail-closed scanner gate now has a bounded automatic exit without
deleting quarantine evidence or relaxing the leader, CAS, movement or
usage-floor fences (rustfs/backlog#2901).

- Recovery retries back off instead of pausing: after the short budget the
  observation reads `backoff`, stays retryable, names the next attempt and is
  capped at ten minutes; one success resets the count and a 24 h window
  restarts the budget (rustfs/rustfs#8297, #8670). The usage-floor failure no
  longer carries the retry count across attempts.
- A corrupt `.bloomcycle.bin` is rebuilt on the iteration after its quarantine
  marker is durable: the fenced leadership claim CAS-replaces the quarantined
  revision under a higher epoch and the marker stays as evidence.
  `future_schema` and `cleanup-pending` remain blocked.
- An unclaimable pause ledger runs the leader degraded: cycles run untracked
  behind the data-movement fence, `persistence_state` reports `degraded`, and
  the claim is retried at most every five minutes (rustfs/rustfs#8251).
- The usage-state full-rebuild reset starts from the empty baseline when every
  usage slot is corrupt and no cycle state exists (rustfs/rustfs#8670, #6951).
- A cycle-state save reconciles non-conflict failures against durable state
  and retries under the same epoch; an exhausted save reloads durable state in
  place. Only a newer owner or an unreadable primary ends the iteration, so
  peers see a changed fence only when ownership changed (rustfs/rustfs#8141).

* test(scanner): align usage-state reset tests with the empty-baseline rebuild

The full-rebuild reset no longer refuses corrupt or identity-less usage slots
when no cycle state exists (rustfs/rustfs#8670). The two tests that encoded
that refusal now check the new contract: the reset starts from the empty
baseline with a bootstrap-pending marker, never an authoritative zero floor,
and a valid backup still supplies the recovery floor.

---------

Co-authored-by: cxymds <cxymds@gmail.com>
2026-10-19 17:49:00 +08:00
Chris 54e39440d2 fix(ecstore): skip node-local prefix dirs in decommission listing (#8702)
Decommission listing failed closed on `.rustfs.sys/buckets/.heal/`, a
directory that holds only the node-local MRF journals on one drive and
no xl.meta. The partial-listing resolver rejected the single prefix
directory below dir quorum and recorded it as unresolved object
metadata, failing the run before the worker's directory skip ran.

Treat a partial sample made only of empty-metadata directory candidates
with zero disk errors as the prefix it is and hand it to the callback
like a quorum-resolved directory; the recursive walk lists any object
beneath it on its own. Samples with disk errors, objects below the
metadata quorum, and mixed samples stay unresolved.

Fixes #8667
2026-10-19 17:48:25 +08:00
cxymds daa3679160 fix(admin): avoid error logs for idle rebalance status (#8694)
* fix(admin): avoid error logs for idle rebalance status

* refactor(admin): reuse rebalance idle error message

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-19 18:03:51 +09:00
Chris 90749fbb06 ci: run feature lanes nightly and warm caches on a schedule (#8683) 2026-10-19 16:18:22 +09:00
Chris e6f76cb64f fix(ci): refresh the e2e-full selection for the new policy tests (#8693)
* fix(ci): refresh the e2e-full selection for the new policy tests

#8615 and #8671 added three e2e tests without refreshing the e2e-full
selection digest, so "Verify e2e full membership" has failed on every
main push since. Regenerate the Linux digest from the listing the failed
run uploaded; the same procedure reproduces the previously committed
value from the last green run's listing, and the name-level diff shows
three additions and no removals. The darwin digest is refreshed
separately from a real macOS listing.

* fix(ci): refresh the darwin e2e-full selection digest

Regenerated from a real `cargo nextest list --profile e2e-full -p e2e_test`
on macOS (rustc 1.99.0, cargo-nextest 0.9.140), so the local check on
macOS agrees with the three policy tests added by #8615 and #8671.
2026-10-19 14:04:23 +08:00
Chris 1ec620ac9c fix(ci): restore the release-profile short-credential guard (#8691)
#8682 replaced the cold `cargo check -p rustfs --release` with a
`cargo rustc ... -C debug-assertions=off` on the rustfs lib alone. That
leaves the dependencies on the dev-profile cfg, so the lib also fails on
items gated on one side of cfg(debug_assertions) only (E0599 on
Filenames::Embedded), and the step's "fails solely at the compile_error"
assertion can never hold; the lane has been red on main and every pull
request since. Restore the whole-graph release check, drop the two-job
limit that only a linking step needed, and record why the single-crate
variant is unsound.
2026-10-19 14:03:57 +08:00
Hauser b3bb9d3596 perf(ecstore): reuse fenced delete settlement inspection
Reuse the fingerprint-validated snapshot while the metadata owner fences an unchanged key. Refresh only after absent-key preparation, retaining destructive preflights, sibling refreshes and final inspection.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-10-19 11:40:51 +09:00
Hauser d156640451 fix(ecstore): recover deletes beside stale data backups
Treat valid foreign transaction markers as inert data-directory leaves during journal-free inspection. Permit fresh authorized missing-key deletion to reclaim safe legacy residue alongside certified absence receipts while preserving automatic ambiguity and generation fences.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
2026-10-19 11:40:51 +09:00
Chris 441b501189 fix(ci): align the ILM build-budget guard with CARGO_BUILD_JOBS 3 (#8687) 2026-10-19 11:40:28 +09:00
Chris c72ed5e909 fix(helm): hold rolling updates until survivors re-admit a replaced pod (#8686) 2026-10-19 11:40:11 +09:00
Chris 392ccadb78 ci: build development packages nightly instead of on every main push (#8684) 2026-10-19 11:39:38 +09:00
Chris f7b2836d7d fix(helm): skip data PVCs when storageclass.dataStorageSize is empty (#8681)
An empty storageclass.dataStorageSize used to be the way to run on
user-supplied drives (extraVolumes/extraVolumeMounts, e.g. JBOD hostPath),
but the chart now always renders data PVC templates, with an invalid empty
storage request.

When the value is empty, render no data volumeClaimTemplates and no data
volumeMounts, drop the data directory preparation from the init container
(and the init container itself when there is nothing left to prepare), and
omit volumeClaimTemplates when no PVC template remains. Standalone mode skips
the data PVC and data volume unless existingClaim.dataClaim is set. Pools may
override the value independently. Default renders are byte-identical.

Fixes #8665
2026-10-19 09:24:00 +09:00
Chris 4d7cd23f8e ci: reuse the dev cache in Connect boundary lanes and raise build jobs (#8682)
Raise CARGO_BUILD_JOBS to 3 on every event in the workspace test lane and
record the closed backlog#1601 experiment: the test-profile compile took
28m21s at 2 on a pull request and 16-23 min at 3 across three successful
main runs.

Merge the two Connect boundary jobs into connect-feature-boundaries. The
offline enrollment gate no longer exports a fresh CARGO_TARGET_DIR, so it
rebuilds only the rustfs crate on top of the restored ci-dev cache; the E2E
root is isolated by the feature gate and the dedicated binary name, not by
the target directory. The short-credential release guard is now
`cargo rustc -p rustfs --lib --features connect-e2e-short-credentials --
-C debug-assertions=off`, which trips the same cfg(not(debug_assertions))
compile_error against cached dev-profile dependencies instead of a cold
release check of the whole graph. Every assertion of both lanes is kept.

Update ci_gate.py and the ci-gates.md PR matrix for the merged job.
2026-10-19 09:16:12 +09:00
Chris 0de680f4a1 chore(docker): upgrade Alpine base image to 3.24.2 (#8680) 2026-10-19 08:57:47 +09:00
Chris ff70c06cef fix(auth): restrict root-derived accounts and credential logs (#8674) 2026-10-19 08:57:31 +09:00
Chris fca83c72e4 ci: move compile-free and prebuilt-binary lanes to hosted runners (#8679)
* ci: move compile-free and prebuilt-binary lanes to hosted runners

Standard GitHub-hosted runners are free for this public repository, while
every sm-standard minute is paid self-hosted capacity. Move the jobs that
never compile (quick checks, typos, scope selection, the required-checks
aggregate, audit, closed-PR cancellation) and the two s3-tests lanes that
only run the prebuilt debug binary to ubuntu-latest, drop the wget install
the custom image needed for typos, and move the upgrade-compatibility
workflow, whose build shape already runs on hosted runners in
e2e-distributed, with renamed cache keys so the hosted and self-hosted
images never share a rust-cache entry. Gate the package summary job on
resolve having run, so development builds stop starting a pod to report a
skipped job.

* docs(ci): document runner billing and placement rules

Record in .github/workflows/README.md which runner labels GitHub bills
(none for this public repository), which ones RustFS pays for, the rule for
when a job may leave ubuntu-latest, the rust-cache single-writer and
same-image constraints, the patterns that have leaked self-hosted minutes,
and a checklist for new or moved jobs. Point to it from .github/AGENTS.md,
docs/testing/ci-gates.md and the ci.yml placement comment so the rule has a
single home.
2026-10-19 08:55:29 +09:00
Chris 22740876b3 docs(architecture): record scanner usage authority and PUT-tail decisions (#8678)
Supersede the 2026-09-03 decision: hard quota admission is owned by the
durable reservation ledger, and scanner data usage is best-effort reporting
input. Record that a pending ordinary PUT commit is a write-barrier event
rather than a usage-publication veto, and split the scanner plan identity
into a coverage identity and a reuse identity.

Refs rustfs/backlog#2872, rustfs/backlog#2903, rustfs/backlog#2904.
2026-10-19 08:33:28 +09:00
Chris ab7001a714 docs: unify agent instructions across tools (#8675) 2026-10-19 07:52:51 +09:00
Hauser b2770312f2 feat(policy): support object creation operation conditions (#8671)
* feat(policy): support object creation operation conditions

Derive the condition from the resolved S3 operation and keep multipart staging separate from object publication. Reject request and claim spoofing and cover policy evaluation and HTTP upload paths.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(protocols): preserve multipart staging authorization

Keep the trusted protocol operation until the shared policy context is built, so FTPS and SFTP honor object creation conditions on multipart staging. Reuse the server-only marker and test real backend decisions for IAM and bucket policies.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): cover ObjectCreationOperation in STS session policies

Run AWS's conditional-write example 1 as an inline AssumeRole session
policy: object publication needs a request header, while multipart
staging is exempt through s3:ObjectCreationOperation.

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: oc7o <oc7o@sloow.de>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-19 07:17:42 +09:00
Chris e0e18908d5 fix(heal): place checkpoint serialization impls before tests (#8663)
* fix(heal): move impls before test module to satisfy clippy items_after_test_module lint

Clippy in Rust 1.99 flags items defined after #[cfg(test)] mod tests.
Move Serialize and Deserialize impls for CheckpointJson before the
test module in crates/heal/src/heal/resume/checkpoint/integrity.rs.

* fix(heal): restore checkpoint integrity file newline

Add the missing final newline so cargo fmt --all --check passes.
The checkpoint serializer implementations and tests remain unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-19 07:08:49 +09:00
Ramakrishna Chilaka 34e883e16a perf(filemeta): avoid clones and intermediate allocations in xl.meta codec (#8661)
* perf(filemeta): avoid clones and intermediate allocations in xl.meta codec

- Slice version headers and metadata payloads directly from the xl.meta buffer in FileMeta::unmarshal_msg instead of allocating temporary header/metadata buffers per version.
- Decode FileInfo directly from borrowed xl.meta slices in get_file_info without constructing an intermediate FileMeta, Vec<FileMetaShallowVersion>, or InlineData copy.
- Validate and search InlineData keys on borrowed slices without per-key Vec/String allocations.
- Borrow FileMetaVersion in FileMetaVersionHeader::from and stream object/delete-marker signatures directly into Xxh64Writer instead of cloning full version structs and marshaling temporary buffers.
- Encode FileMetaVersionHeader into a stack buffer in FileMeta::marshal_msg and pre-size output buffers.
- Decode msgpack struct field names onto a 16-byte stack buffer via read_small_str and unwrap PrependByteReader after reading submap headers.

* test(filemeta): pin xl.meta codec compatibility

Pin pre-optimization signatures for historical RustFS and MinIO fixtures,
and cover direct reads across selectors and decode options. Preserve null
inline keys, free-version selection, empty histories, and allocation
boundaries for large version bodies.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-19 07:01:11 +09:00
0x70 3a5742c5fd feat(policy): support s3:if-none-match and s3:if-match condition keys (#8615)
* feat(policy): support s3:if-none-match and s3:if-match condition keys

Bucket, IAM, and STS session policies that enforce conditional writes
with s3:if-none-match or s3:if-match were rejected with "invalid key
name". Accept both keys and populate them from the If-None-Match and
If-Match request headers. A blank header is left out of the condition
values because the write ignores it, so it cannot satisfy a Null check
for a write that runs unconditionally.

* fix(policy): classify conditional write keys as request-derived

s3:if-match and s3:if-none-match are filled from request headers, but
is_server_derived() reported them as server-derived because their names
lack the x-amz- prefix. Add an explicit arm so header input is never
refused for these keys if they join COMMON_KEYS.

* fix(s3): reject non-wildcard If-None-Match on writes

S3 accepts If-None-Match on PutObject, CopyObject and
CompleteMultipartUpload only as "*" and answers other values with 501
NotImplemented. RustFS compared such a value as an ETag, so a write with
If-None-Match: "<other etag>" overwrote the key while satisfying a
policy that requires s3:if-none-match.

Reject ETag values in the three handlers before any I/O. The store-level
check now treats any non-blank If-None-Match as create-only, so a caller
that skips the handler check cannot weaken the precondition. Drop the
CompleteMultipartUpload ETag pre-check that the rejection supersedes.

* test: tighten conditional write policy coverage

Pin trimming and the non-UTF-8 case in the condition values, and drop
the auth test that re-ran policy evaluation already covered by the
policy crate. Run the three policy e2e scenarios on one server, assert
that ETag If-None-Match cannot overwrite under a Null: false policy and
that CompleteMultipartUpload with If-None-Match: * returns 412. Share
the AssumeRole response parser through common.rs.

* test(ecstore): isolate manual delete recovery from workers

Stop fixture background services before seeding recovery journals so the
explicit recovery pass owns the decisions its assertions inspect.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): enforce request-derived conditional write guards

Validate raw If-None-Match headers as well as DTO fields so POST form
uploads cannot skip rejection of unsupported ETag preconditions. Derive
conditional policy keys after identity claims and clear colliding claims
when the actual request has no usable precondition.

Align multipart and pinned Ceph regressions with wildcard-only writes,
retaining If-Match, object-content, version and delete-marker assertions.
Document that 501 is the implementation's unsupported-value response.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(s3): eliminate conditional write request copies

Borrow request headers during policy condition collection and exclude
conditional keys at the identity-claim boundary instead of inserting and
removing them. Preserve existing aliases and multivalue header behavior.

Move the CompleteMultipartUpload input out of its request, compare strong
ETags by reference, and remove the ETag If-None-Match branch made unreachable
by the write guard. Keep both form/HTTP input checks and storage commit
preconditions intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(s3): align ignored conditional PUT and COPY cases

Use If-None-Match: * for the existing-object 412 regressions and assert 501
for matching and nonmatching ETag values. Cover both header-only callers
and parsed DTO-plus-header requests in the serial ILM fixtures.

Check object bytes and live version counts after each rejected operation.
Keep the existing test names and ignored-test lane selection unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-19 07:00:21 +09:00
InitCool e2583ddcd6 feat(oidc): make the Console session duration configurable (#8568)
* feat(oidc): make the Console session duration configurable

The OIDC callback hard-coded the Console credential lifetime at 3600
seconds, so a Console session always ended after about an hour.
RUSTFS_CONSOLE_AUTH_TIMEOUT, the documented knob next to this behaviour,
is the Console HTTP request timeout and cannot extend a session.

Resolve the lifetime from the process-level
RUSTFS_CONSOLE_OIDC_SESSION_DURATION instead and forward it to
complete_authorization_code(). Unset, unparseable, zero, and negative
values keep DEFAULT_CONSOLE_OIDC_SESSION_DURATION_SECS (3600), so
existing deployments keep the previous behaviour and a misconfigured
variable can never issue an already expired credential. Values above
CONSOLE_OIDC_SESSION_DURATION_MAX_SECS (7 days) are clamped to that
maximum with a warning, which also caps how far an operator can stretch a
credential by accident.

Keep the Console logout token for that same lifetime. OidcStateStore
pinned the logout cache to a hard-coded hour, so a session configured to
last longer than that dropped its logout token while the credential was
still valid: RP-initiated logout could no longer resolve, and the browser
was sent back to the Console login page with the IdP session still
active. The logout cache now derives its TTL from the same
console_session_duration_secs(), which applies the same clamp, and the
callback reads its lifetime from that one function instead of a local
copy, so the credential lifetime and the logout-token retention cannot
drift apart. Add tests covering the default hour, an extended session,
the maximum, and out-of-range values.

The variable is deliberately not part of ENV_IDENTITY_OPENID_KEYS: that
list is suffix-scanned per provider, and this setting applies to every
provider. It has no identity_openid KVS equivalent and needs a restart,
like the other process-level settings in the same document.

Document the setting in the login flow, the process-level table, and the
troubleshooting table, and clarify that RUSTFS_CONSOLE_AUTH_TIMEOUT is a
request timeout rather than a session lifetime.

* test(ecstore): retain offline marker fixtures across reconnect

Keep simulated drive handles attached and reject I/O until explicit rejoin.
Attempt reconnect in the current-marker case so the former CI race is
deterministic while retaining physical cleanup and historical-data checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(oidc): align session diagnostics and lifetime regression tests

Use structured IAM diagnostics for clamping and move configuration tests
to their owning crate. Cover duration boundaries, logout retention and
signed credential expiration through the federation binding.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-19 06:52:05 +09:00
Henry Guo 317191ce00 fix(scanner): scale backup deadline to checkpoint save cost (#8659) 2026-10-19 06:21:33 +09:00
Chris f3c4a790dd fix(listing): accept literal prefixes with repeated slashes (#8605)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-19 01:27:51 +08:00
Hauser 6681dd12e3 fix(scanner): expose mixed sweep checkpoint transitions (#8657)
* fix(scanner): expose mixed sweep checkpoint transitions

Correlate checkpoint admission, incomplete sweeps, and successful terminal
Partial persistence by bucket/source and cycle/leader epoch. Share the
traversal-position predicate and local/remote save diagnostic without
changing certification, acknowledgement, or persistence contracts.

Extend real-disk cache regressions across cycles and leader changes and
verify that failed terminal saves retain an older periodic frontier until
a successful terminal save permits fresh certification.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): report the aggregate incomplete sweep gate

Expose the existing unresolved_objects decision in the incomplete-sweep
event. Nested cache-entry failures can keep a sweep partial while both
the root counter and the capped diagnostic failure map are empty.

Extend the real-disk regression with diagnostics disabled and distinguish
this case from mixed-only coverage without adding another cache walk.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-18 16:50:42 +08:00
Henry Guo 3b0ee83466 fix(scanner): pace checkpoints by snapshot construction cost (#8656) 2026-10-18 13:36:58 +08:00
Henry Guo 02c4f4c97f fix(ecstore): clean completed PUT staging on pre-commit cancellation (#8654)
fix(ecstore): clean staged PUT data on pre-commit cancellation
2026-10-18 13:36:33 +08:00
Hauser c28e02a722 fix(ecstore): preserve inline heal rollback staging lifetime (#8650)
* fix(ecstore): preserve inline heal rollback staging lifetime

Retain the TMP source prefix in the existing publication lease until the owned receiver finishes rollback. Serialize recursive cleanup and stale GC with the same key, and avoid decoding an unused old version during inline healing.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reuse staging lease during parent cleanup

Reuse the existing staging lease when pruning its parent directory, including already-missing prefixes, and retain it in the physical rmdir executor after cancellation. Match the inline-heal fsync test hook to the canonical group path on Linux.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-18 12:58:57 +08:00
Henry Guo 1df96624df feat(table-catalog): support conditional maintenance worker retries (#8562)
* feat(table-catalog): support conditional maintenance worker retries

* fix(table-catalog): refresh conditional worker claim time

* style(table-catalog): format conditional worker expiry test

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-18 10:03:58 +08:00
Henry Guo ee534fa19b test(table-catalog): validate current backing contracts (#8559)
* test(table-catalog): qualify current backing contracts

* fix(table-catalog): verify smoke cleanup after object rename

* fix(ci): include the baseline writer test in full E2E selection

* fix(ci): reserve test capacity for disk-backed lock probes

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-18 10:03:30 +08:00
Chris 668bc7a4ba test(utils): cover mixed-case metadata alias conflicts (#8651) 2026-10-18 10:02:47 +08:00
Hauser e29a76f391 fix(heal): verify checkpoint integrity before applying defaults (#8649)
* fix(heal): verify checkpoint integrity before applying defaults

Validate the persisted JSON shape so intact signed legacy checkpoints can
reach the existing safe schema rewind. Preserve deployed field ordering and
read historical sorted-key digests without changing the checkpoint schema.
Reuse the signing value when saving and retain strict duplicate-field parsing.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): preserve checkpoint writer rollback compatibility

Keep ordered checkpoint parsing local rather than enabling preserve_order
throughout the server. Sort writer keys recursively to retain the production
base reader's digest format while accepting both historical read orders.

Cover real writer canonicalization, nested scalar ordering and depth limits.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-18 04:03:16 +08:00
Chris 3ddd307304 chore(release): align installation references for 1.0.3 (#8646)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-18 02:58:32 +08:00
Ramakrishna Chilaka 9946c67b53 perf(filemeta): avoid clones and redundant map scans in into_fileinfo (#8647)
FileMetaShallowVersion::into_fileinfo and to_fileinfo_with_part_checksums previously decoded an owned FileMetaVersion and then converted it via shared reference (&self), cloning meta_user, meta_sys, part_etags, and part_indices before dropping the temporary FileMetaVersion and scanning the metadata map up to 10 times for optional internal suffixes. Additionally, metadata_compat helpers (get_str, contains_key_str, remove_str, get_consistent_value) allocated two temporary heap Strings via both_keys(suffix) on every absent-key fallback.

- Replace both_keys(suffix) in case-insensitive fallback scans with allocation-free prefix+suffix slice matching.
- Add consuming into_fileinfo_with_part_checksums on FileMetaVersion and MetaObject to move decoded part vectors and metadata maps in place.
- Fold transition-version metadata formatting and optional suffix presence detection into the single meta_sys pass, and skip target_delete_marker_versions in get_internal_replication_state when no delete-marker version keys are present.
2026-10-18 02:57:21 +08:00
Hauser 286e4cf30b perf(ecstore): keep inline heal rollback backups in staging (#8648)
* perf(ecstore): keep inline heal rollback backups in staging

Keep inline repair rollback local to the disk commit and avoid publishing
a live backup that needs a second cleanup operation. Preserve legacy peer
and non-inline backup reclamation, with production-mode rollback coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Co-Authored-By: Ramakrishna Chilaka <rkchilaka378@gmail.com>

* test(ecstore): route inline heal imports through test facade

Keep ECStore integration-test API access behind storage_api.rs to satisfy
the architecture migration boundary without changing test behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Ramakrishna Chilaka <rkchilaka378@gmail.com>
2026-10-18 01:51:52 +08:00
Hauser 453be3dd84 fix(scanner): attribute incomplete census and lease waits (#8643)
* fix(scanner): attribute incomplete census and lease waits

Report bucket sweep blockers and correlate partial remote work and rejected
roots with their set and cycle. Keep distinct accepted bucket names in the
result collector so completion counts and missing-bucket attribution share
one source of truth, including during cancellation drain.

Distinguish worker completion from coverage, retain peer/purpose context in
lease failures without changing retry suffixes, and avoid claiming that an
unverified leader fence changed or a waiting scan was complete.

Document the census stages and preserve publication, dirty acknowledgement,
remote failure classification and checkpoint safety rules.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(scanner): move set caches through the result collector

Give the sole result collector exclusive ownership of the aggregate cache
and return it together with the accepted bucket names after channel drain.
Move that cache into complete persistence or incomplete updates instead of
sharing a mutex and cloning the terminal cache.

Return each accepted result's owned name from cache application so coverage
tracking also avoids an extra name allocation. Preserve worker joins,
cancellation drain, publication metadata and all certification fences.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-18 01:05:14 +08:00
唐小鸭 222beaba00 fix(scanner): certify dirty usage per bucket and complete cycles with pending buckets (#8542)
* fix(scanner): treat replication status writebacks as status-class hits

A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).

The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.

No persisted format, protocol, admin API or configuration changes.

This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.

Refs #8375, #8494, rustfs/backlog#2816.

* fix(scanner): refresh replication statistics during maintenance

Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): avoid expanding the legacy S3 DTO surface

Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(scanner): attribute activity guardrail vetoes per field

The post-walk cluster activity proof compared the whole activity snapshot,
so any namespace or dirty-usage generation movement vetoed publication.
Veto only on the fields that change which heap the walk described
(peer set, instance id, protocol version, topology, maintenance
generation, data movement, movement generation, publication blocked) and
name the field and host in the log and in a new usage-freshness counter.
Generation movement is a shade signal consumed per bucket. A remote peer
whose dirty-usage state moved keeps the whole-cycle outcome until scoped
acknowledgements can name its buckets.

* feat(data-usage): carry pending-bucket certification in DataUsageInfo

Add the additive fields a per-bucket certified scanner snapshot needs:
the sorted list and count of buckets whose dirty-usage generation moved
during the walk, and an explicit observational marker for snapshots whose
authoritative publication was vetoed. All three default when absent and
are skipped when empty, so older readers see the existing snapshot shape.

* feat(scanner): certify dirty usage per bucket and complete cycles with pending buckets

A structurally complete walk that the guardrail did not veto is now
`Complete` even when the dirty-usage generation moved during the walk.
Certification keeps the compare-and-remove semantics of
clear_dirty_usage_buckets: buckets whose generation still equals the start
snapshot are cleared, buckets written during the walk stay in the dirty map
and are published as the pending set, so usage_snapshot_converged becomes
`pending == 0`. The authoritative object is written for such cycles; the
observed route is selected by the explicit observational mark that vetoed
cycles carry instead of by the convergence flag, and the root publication
proof is read for every authoritative save. The post-lease activity recheck
compares the guardrail fields, with the peer dirty-usage fallback, instead
of the full snapshot digest.

* feat(scanner): accept unconverged complete publications as the cache baseline

complete_scanner_cache_baseline_plan_digest required the authoritative
snapshot to be converged, so a site whose cycles always left a bucket
pending could never scope a rescan and every cycle was a full walk. The
authoritative object is now a structural baseline whenever it carries a
convergence marker and no observational mark; bucket-cache reuse is then
governed by certification, because pending buckets stay in the dirty map
and are selected for the rescan while certified buckets reuse their set
cache. Observed candidates keep requiring the unconverged marker, and
segment reuse evidence keeps requiring a converged baseline.

Add the single-disk cycle test: a bucket written during the walk is the
only pending bucket, the others are certified, the publication is
authoritative and unconverged, and the next cycle is scoped to that
bucket; a held PUT tail still supersedes onto the observed route.

* feat(admin): expose scanner guardrail vetoes in usage freshness

Carry the attributed guardrail veto counter, timestamp and <field>@<host>
reason from the scanner metrics report into the madmin scanner status so
an operator can tell a rebalance or a PUT tail from a broken probe. The
fields are additive with serde defaults and merge like the deferral
fields.

* docs(scanner): record per-bucket certification in the design and changelog

Amend the convergence design where the implementation deviates: pending
buckets carry their walked value, maintenance_generation stays a veto,
guardrail localisation and the remaining phase-2 items ship separately,
last_cycle_result keeps its success label, and the admin surface gains the
guardrail veto fields. Update the publication contract for the post-lease
guardrail recheck and the explicit observational routing mark.

* refactor(scanner): drop the unreachable failed-bucket input from the pending set

A failed bucket never reaches a Complete cycle, so the pending set only
needs the start snapshot. Also remove a redundant clone in the data-usage
round-trip test.

* test(scanner): name the activity field mutation table type

* test(scanner): fill the post-walk gate in merged replication fixtures

The replication maintenance fixture merged from main builds a
ScannerCycleRequest literal and predates the test-only post_walk_gate
field; give it the default so the test target compiles on the rebased
branch.

* ci(nextest): run the admin target-repair fixtures on the ecstore base stack

repair_transaction_rejects_a_bucket_recreated_after_target_validation
overflowed the 2 MiB libtest default twice in a row on Linux CI for this
branch. Measured by sweeping RUST_MIN_STACK with the ci profile on macOS
arm64: main (65f30bd5c) passes at 2032 KiB and overflows at 2016 KiB,
this branch passes at 2040 KiB and overflows at 2032 KiB; on both trees
the overflow is in the test's delete_bucket step on the real store, and
the futures on that path are small (the peak is nested poll depth, not
one large future). The module was given this stack in #7272 and removed
in #7280 to check that boxing (#7277) sufficed; with the default now
exhausted on main itself, give the fixtures the same 4 MiB base stack as
the other real-storage rustfs fixtures, in both profiles.

* test(scanner): fill the post-walk gate in the walkers-per-disk fixture

The merge from main brought the walkers_per_disk cycle fixture (#8539),
whose ScannerCycleRequest initializer predates the test-only
post_walk_gate field this branch adds; the scanner test target did not
compile. Fill the default so the merged head builds.

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-18 00:25:02 +08:00
Hauser aeae6f9980 fix(ecstore): reclaim rollback backups after object heal (#8641)
* fix(ecstore): reclaim rollback backups after object heal

Retire only the backup returned by a successful, fenced heal publication.
Keep child keys, live data directories, and delete recovery evidence safe.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): retain heal cleanup ownership through cancellation

Forward publication and repair key owners to the existing owned delete
executor so local unlink cannot outlive its exact-key serialization.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-18 00:03:00 +08:00
Hauser f64cb816db fix(scanner): preserve checkpoint progress across object failures (#8642)
Separate completed traversal from persisted object health debt so leader
handoffs reuse healthy subtrees and retry failures without losing progress.
Fence old readers with a new receipt digest domain and preserve exact
accounting under diagnostic eviction, compaction and punctuation ordering.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-17 23:33:47 +08:00
Hauser 816e18f2c8 fix(filemeta): read large metacache records incrementally (#8640)
* fix(filemeta): read large metacache records incrementally

Replace the aggregate 16 MiB guard with fallible chunked growth so
versioned xl.meta records remain listable without trusting length prefixes.
Discard skipped payloads with a bounded buffer and distinguish truncated
records from compatible empty or between-entry EOF.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(filemeta): remove redundant metacache reader copies

Track the returned entry with a boolean instead of cloning its metadata,
decode object names directly from the borrowed slice, and derive the read
position from buffer length. Preserve scratch reuse, exact output capacity,
wire framing, and sticky corruption errors.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-17 23:25:21 +08:00
Chris e98992bb87 docs(release): add post-release advisory reconciliation (#8639)
* docs(release): add post-release advisory reconciliation

* docs(release): remove hard wraps from advisory guidance
2026-10-17 22:01:08 +08:00
cxymds ce0c532740 fix(lifecycle): reject expiry from stale policy snapshots (#8637)
* fix(lifecycle): reject expiry from stale policy snapshots

* fix(lifecycle): keep expiry snapshots behind domain boundary

* test(ecstore): fix lifecycle fixtures without test-util
2026-10-17 21:45:30 +08:00
Chris 3defc19d5e fix(ci): refresh E2E selection and isolate disk lock probes (#8638)
* fix(ci): include the baseline writer test in full E2E selection

* fix(ci): reserve capacity for short-deadline disk lock probes
2026-10-17 21:04:04 +08:00
cxymds b1f3ebb184 fix(ecstore): converge partial pool identity writes (#8636)
* fix(ecstore): converge partial pool identity writes

* fix(ecstore): bound pool metadata CAS future state
2026-10-17 20:45:49 +08:00
唐小鸭 b40f1e5fab fix(utils): harden Win32 volume probes and flag Windows network drives (#8631)
* fix(utils): NUL-terminate and extend paths passed to Win32 volume APIs

`get_volume_name` returned the volume root truncated at its NUL and the
result was then passed to `GetVolumeInformationW` as a PCWSTR, an
out-of-bounds read. The disk-space and volume APIs also received the
caller's raw path, so they failed beyond MAX_PATH, and a UNC share root
lacked the trailing backslash these APIs require.

- Wrap every Win32 path in a NUL-terminated `WidePath` that rejects
  interior NULs; volume roots returned by `GetVolumePathNameW` keep
  their terminator.
- Convert inputs to the extended-length `\\?\` / `\\?\UNC\` form after
  `GetFullPathNameW` normalization, and give directory queries a
  trailing backslash.
- Query `GetDiskFreeSpaceW` and `GetVolumeInformationW` on the volume
  root as documented; size buffers to the documented MAX_PATH + 1 or
  the input length.
- Report Win32 error codes instead of raw HRESULTs so `ErrorKind`
  matches `std::fs`.
- Keep volume ids in their conventional `C:\` / `\\server\share\` form.

Add Windows-only `os::` tests for long directories, UNC share roots and
non-empty fstype, and run them in the Windows filesystem workflow.

* fix(utils): report Windows network volumes to the fs guard

Windows names the server's filesystem (usually NTFS) for SMB shares, so
the startup unsupported-filesystem guard, which matches fstype names
such as cifs or smb2, never fired for UNC paths or mapped drives.

- Add `DiskInfo::remote`. On Windows it is set when the directory handle
  answers `FileRemoteProtocolInfo` (only network redirectors do, and the
  handle follows symlinks and junctions to their target) or the volume
  root's `GetDriveTypeW` is `DRIVE_REMOTE`. Unix keeps it false because
  the client fstype already names network mounts.
- Treat remote volumes as unsupported in the startup guard and label
  them "<fstype> on network share".
- Cover UNC shares, paths beyond MAX_PATH below a share, a directory
  symlink to a share and a `net use` mapped drive with Windows-only
  `os::` tests that skip when `\\localhost\<drive>$` is unreachable;
  the Windows filesystem workflow maps the drive before running them.

* chore(deps): replace yanked inout 0.2.3 with 0.2.4

cargo-deny's advisories check rejects the yanked inout 0.2.3 that main's lockfile pins, failing every open PR. Lockfile-only patch bump.

* fix(utils): detect network targets from the final NT path

On the Windows runner, a directory symlink to \\localhost\C$ was reported as local: FileRemoteProtocolInfo did not identify it, and the volume root of the link is a fixed drive.

Also treat a handle whose final NT path (GetFinalPathNameByHandleW with VOLUME_NAME_NT) lies under \Device\Mup\ as remote; the Multiple UNC Provider fronts every UNC and mapped-drive target. The tests now assert the handle probe on its own for local and share paths and report both signals when it disagrees.

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-17 19:10:09 +08:00
唐小鸭 276cb24e26 fix(ecstore): share cache and lock identity across Windows key aliases (#8635)
* fix(ecstore): share metadata cache identity across Windows key aliases

Windows disks treat `\` as a path separator and `path_join_buf` drops the
empty segments that leaves, so `a\b`, `a\\b`, `\a\b` and `a\b\` all
address the xl.meta of `a/b`. The GetObject metadata cache keyed and
hashed the raw spelling, so DELETE `a\b` retired nothing cached for
`a/b`: GET/HEAD `a/b` kept returning the deleted object for up to the
2s cache TTL.

Fold the object name into the same segment form the disk resolves to
before computing the cache hash, key and lookup key. Off Windows the
name stays verbatim and borrowed, so `\` remains an ordinary key byte.

* fix(ecstore): serialize Windows backslash key aliases on one namespace lock

Windows disks treat `\` as a path separator and drop the empty segments
it leaves, so `a\b`, `a\\b`, `\a\b` and `a\b\` all address the xl.meta
of `a/b`. Namespace locks keyed the raw spelling, so writes, deletes and
heals through different spellings ran concurrently on one xl.meta and a
read-modify-write in rename_data could drop a version.

Fold the object name into its disk identity (rustfs_utils::path::
object_name_disk_identity, a no-op off Windows) wherever a namespace
lock key is formed:
- SetDisks::new_ns_lock, which every object lock path goes through;
- the batch delete lock request and its dedup/lookups, which build
  ObjectKeys directly.

Two consumers compared lock identities by spelling and follow along:
- WriteCommitGuard records the folded key from the lock resource, so
  matching a borrowed guard now folds the target name too;
- CopyObject holds the source read guard until EOF on streaming reads.
  `a\b\` -> `a/b` is not a same-object copy by path_join_buf, but shares
  one lock, so it now takes the self-copy lock instead of waiting on
  itself until the lock timeout.

Distributed lock keys are folded by the requesting node only; lock
servers compare them verbatim.

* refactor(ecstore): key the metadata cache by the shared disk identity

The metadata cache fix carried its own copy of the Windows backslash
folding rule. Use rustfs_utils::path::object_name_disk_identity, which
namespace locks already use, so cache and lock identities cannot drift.

* ci(windows): run filesystem tests when key identity rules change

The Windows backslash folding for object names lives in utils path.rs
and is consumed by the namespace lock and write commit guard; run the
windows_ tests when any of them changes.
2026-10-17 18:06:29 +08:00
cxymds 508caab281 fix(lifecycle): skip empty delete replication tasks (#8628)
* fix(lifecycle): skip empty delete replication tasks

* test(lifecycle): route replication hooks through the sink

* test(e2e): make upgrade write readiness checks deterministic

* fix(deps): replace yanked inout 0.2.3 with 0.2.4
2026-10-17 17:15:37 +08:00
唐小鸭 776cdc517b fix(ecstore): keep Windows disk roots out of verbatim form (#8633)
* fix(ecstore): keep Windows disk roots out of verbatim form

The disk root comes from GetFinalPathNameByHandleW, which returns a
verbatim path. `simplified` only strips `\\?\C:` below MAX_PATH, so a
data directory past 260 characters (or one reached through a UNC share)
kept the prefix. Disk IO extends the root by joining `/`-separated
relative paths as strings (`read_raw` builds `<object>/xl.meta` with
`path_join`), and inside a verbatim path `/` is not a separator: the
first metadata read, pool.bin.identity, failed with ERROR_INVALID_NAME
until startup gave up.

Add `rustfs_utils::non_verbatim`, which strips the disk and UNC verbatim
prefixes without a length limit while still keeping paths whose
components Win32 would rewrite, and use it for the publication root.
std re-adds the extended-length prefix for long paths.

Cover a root past MAX_PATH with a disk-level read/write test and run all
`dunce::` tests in the Windows filesystem workflow.

* fix(ecstore): query Win32 volume APIs with extended long disk roots

A disk root past the legacy limit used to reach `rustfs_utils::os` in
verbatim form, which the Win32 volume APIs accept. Now that the root
stays non-verbatim, GetDiskFreeSpaceExW rejected it with
ERROR_PATH_NOT_FOUND and LocalDisk::new failed in the Windows filesystem
workflow. Pass long roots to `get_info` and `get_physical_device_ids` in
extended-length form; shorter roots stay unchanged so reported volume
identities keep their `C:\` form.
2026-10-17 17:04:21 +08:00
cxymds 8dd399cd58 fix(lifecycle): compact inline versions once per deletion batch (#8634) 2026-10-17 16:16:30 +08:00
唐小鸭 353e67bbf3 fix(ecstore): resolve Windows drive paths in URL endpoints (#8629)
On Windows, URL endpoints such as http://host:9000/C:/data{1...4} (the only
form multi-node and pool-expansion layouts accept) produced the disk path
`/C:/data1`: `try_from` stripped the synthetic slash, but `Url::set_path`
put it back, and `get_file_path` only removed it for `file` endpoints.
Windows resolves `/C:/data1` as an invalid `C:` component under the current
drive, so disk validation and `resolve_local_disk_root` failed.

- `get_file_path` now restores `/X:/...` to `X:/...` for every scheme on
  Windows; URL paths without a drive (`/export1`, `/C:foo`) are unchanged.
- UNC shares (`\\nas\share\d1`, `//nas/share/d1`, `\\?\UNC\...`) are rejected
  with an explicit error instead of becoming a file URL whose host is the
  share server and whose path drops the share name.
- A drive root (`http://host:9000/D:/`) is accepted in URL form, matching
  the path-style `D:\` endpoint that was already accepted.
- The "distinct physical disks" error explains on Windows that directories
  on one volume share a device identity, alongside the existing
  RUSTFS_UNSAFE_BYPASS_DISK_CHECK hint.
- The Windows filesystem lane now runs on layout changes and runs the
  layout::endpoint:: tests; new Windows tests carry the windows_ prefix.
- Document Windows endpoint forms, the same-volume check, and the SNSD
  expansion limit in cluster-lifecycle-operations.md.
2026-10-17 16:06:37 +08:00
唐小鸭 78ea655d87 fix(startup): drive startup on an explicitly sized thread (#8630)
`run_process` called `block_on(async_main())` on the native main thread,
so the whole startup chain ran on whatever stack the platform hands the
main thread: 8 MiB from `ulimit -s` on Linux and macOS, but the 1 MiB PE
default on Windows. A Windows debug build overflowed before it listened.
`RUST_MIN_STACK` does not reach the native main thread.

Run the runtime on a `rustfs-main` thread with an explicit 8 MiB stack,
keeping the budget Linux and macOS already had and making it independent
of the linker, toolchain (msvc/gnu) and `ulimit`. Panics are re-raised
on the main thread so the exit status is unchanged.

Measured on macOS arm64 by lowering the main-thread stack with
`ulimit -s` before this change: debug startup overflows at 1792 KiB and
reaches /health/ready at 2048 KiB; release overflows at 512 KiB and is
ready at 768 KiB. After it, debug starts with a 256 KiB main thread.

Add a Windows `Startup Smoke` job that builds the debug server, runs
`--version`, and waits for /health/ready on a single volume.
2026-10-17 16:06:19 +08:00
唐小鸭 5efae5baa6 fix(ecstore): unify object key admission for Windows hosts (#8632)
* fix(ecstore): admit CopyObject destinations like PutObject

Object-key platform checks were split across call sites with gaps:
CopyObject only ran check_bucket_and_object_names on its destination and
then reached the pools' put_object directly, so it admitted keys PutObject
rejects (over 1024 bytes, leading slash, and on Windows reserved characters
or unaddressable segments, the "writes but cannot be read" case of #3449).

- rustfs-utils: expose check_windows_file_name as the single per-segment
  Win32 rule set; dunce's simplified() reuses it. Reserved device names now
  include COM/LPT with superscript digits and CONIN$/CONOUT$.
- ecstore: replace the duplicated reserved-name table with that function;
  Windows also rejects control characters 0x01-0x1F (NUL keeps its
  platform-independent ObjectNameInvalid answer).
- ecstore: add check_new_object_args as the shared admission entry for
  PutObject, cross-key CopyObject destinations and CreateMultipartUpload.
  In-place copies (metadata replace, restore) keep the copy-only check so
  existing keys stay rewritable. CreateMultipartUpload now also rejects
  over-NAME_MAX segments up front instead of failing at completion.
- CI: run bucket::utils and dunce tests in the Windows filesystem job.

* fix(ecstore): reject new backslash keys on Windows hosts

A Windows disk treats '\' as a path separator, so a new key `a\b` lands on
the same xl.meta as `a/b` and the two keys overwrite each other.

Reject '\' only where a key enters the namespace (check_new_object_args:
PutObject, cross-key CopyObject destinations, CreateMultipartUpload) and
only on Windows. Reads, deletes, in-flight multipart operations, in-place
copies, data movement and internal meta buckets keep accepting it, so keys
already stored with '\' stay readable, deletable and movable.

Pending owner ruling: whether new '\' keys should be rejected at all, and
whether reads/deletes should keep accepting them.

* ci(windows): quote filtered test runs and drop duplicate path
2026-10-17 15:53:16 +08:00
Henry Guo 8110ab2893 fix(admin): preserve dirty usage over observed snapshots (#8627)
fix(data-usage): retain dirty counts over observations

Preserve unreconciled local mutations when an observed snapshot completes later. Add focused selection-to-overlay regressions for writes, deletes and authority boundaries.

Validation checkpoint: the pre-fix regression failed as expected. Post-fix tests, formatting and runtime verification were not run before the user stopped the task. This local commit is not ready for publication.
2026-10-17 15:10:11 +08:00
Ramakrishna Chilaka 59ea8bf8ce fix(s3): return 412 for conditional multipart uploads on existing keys (#8625) 2026-10-17 14:09:04 +08:00
Henry Guo 749a1686aa feat(table-catalog): plan maintenance on durable backing (#8567)
* feat(table-catalog): plan maintenance on durable backing

* test(table-catalog): bootstrap strong planning handler fixture
2026-10-17 12:06:02 +08:00
Chris 6db3158740 fix: resolve clippy warnings from Rust 1.99 lints (#8619)
- Combine identical if-block arms in runtime_sources.rs (clippy::if_same_then_else)
- Allow redundant_locals for cfg-gated closure captures in metadata_read.rs and read.rs
- Remove needless borrow in upgrade_compatibility_test.rs (clippy::needless_borrows_for_generic_args)
2026-10-17 11:46:33 +08:00
Henry Guo c07a5d292c fix(connect): retain bounded object cleanup diagnostics (#8624) 2026-10-17 10:57:39 +08:00
overtrue 99abffc57a refactor(lifecycle): replace s3s domain types with Gateway DTOs
Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.
2026-10-17 07:03:41 +08:00
overtrue fc061c7930 fix(ecstore): retry tier commit intent lock contention
Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.
2026-10-17 01:19:11 +08:00
Chris 00d1f6ead2 chore(release): align installation references for 1.0.2 (#8617) 2026-10-16 23:19:34 +08:00
Henry Guo adfe60ec7d fix(connect): prepare profile fixtures before timed IPC (#8616) 2026-10-16 23:16:19 +08:00
Henry Guo a5c6f6195e fix(scanner): retain partial progress without a usage update (#8612) 2026-10-16 21:05:33 +08:00
Ramakrishna Chilaka ec0208480e fix(ecstore): publish delete rollback files via verbatim-safe rename (#8607) 2026-10-16 20:04:44 +08:00
overtrue d86bf1360f fix(ecstore): recover delete batches with missing versions
Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
2026-10-16 17:32:35 +08:00
唐小鸭 62b6b6149c fix(ecstore): publish delete rollback backups past Windows MAX_PATH (#8606)
tempfile's Windows `persist` passes the raw paths to MoveFileExW and
SetFileAttributesW, bypassing std's extended-length handling. Once
`<object>/<rollback>/xl.meta.bkp` reached 260 UTF-16 units every delete
of that key failed with 500, while PUT and GET kept working.

Add `rustfs_utils::extended_length_path`, which normalizes a path with
GetFullPathNameW and prefixes it with `\\?\` (or `\\?\UNC\`), and use it
for the staging directory and destination handed to tempfile. Cover the
single and batch delete paths with a deep-key regression test and run the
helper's Windows assertions in the Windows filesystem workflow.

Fixes #8545
2026-10-16 16:16:32 +08:00
唐小鸭 741de6f81c fix(replication): locate encrypted replicas without comparing ETags (#8602)
Since 1.0.2 an encrypted object's ETag is a keyed fingerprint, not the
plaintext MD5. A target that computes its own ETag (AWS S3, Wasabi) never
reports it, so every "find the existing replica by exact ETag" step missed
encrypted objects on targets that mint their own version ids:

- a heal / MRF re-drive after a lost PUT response wrote a second version;
- an existing-object resync re-PUT a replica its ledger already named
  (the ledger-located HEAD was confirmed by ETag only), and on the AWS
  400 shape the ledger was not consulted at all;
- resync verification counted such replicas as failed;
- a ledger-less version purge reported the replica as already gone.

Encrypted sources now recognise their replica by shape: the byte count the
replication PUT sent (plaintext for managed SSE, ciphertext for SSE-C) and
a target write time no earlier than the source version. An ETag match
still wins (RustFS/MinIO targets). The ledger is consulted first on both
the 404 and the AWS 400 shape. Without a ledger, only a unique shape match
among the unclaimed ListObjectVersions candidates is adopted and recorded;
several shape matches mean re-PUT for convergence and a refused purge. The
AWS current-version fallback stays ETag-only, and a listing the
target refuses on the 400/403 shape keeps that fallback (re-PUT) rather
than stalling. An unversioned target's `null` version is never matched by
shape: it holds whichever generation was written last. Plaintext sources keep exact
ETag identity.

Builds on the ETag-computing fake target mode and the ComputesOwnEtags
matrix row from rustfs#8583. Adds an SSE-S3 variant of the mint-own-ids
re-drive test (red before this change: the heal left two target versions),
and both variants now also assert the purge.
2026-10-16 13:44:33 +08:00
Chris f5ff8783f4 fix(ecstore): box large enum variant in CurrentDeleteGeneration (#8576) 2026-10-16 13:11:49 +08:00
overtrue f6097d1df2 fix(ci): restore integration test compilation
Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.
2026-10-16 13:06:24 +08:00
cxymds 12ae143bee fix(quota): keep writes available with a degraded fleet (#8604) 2026-10-16 13:05:20 +08:00
cxymds 628663f468 fix(heal): recover replacements after pool removal (#8600)
* fix(heal): recover replacements after pool removal

* test(e2e): gate upgrade probes on writable baseline nodes

* test(e2e): register baseline readiness regression in smoke digest
2026-10-16 12:08:29 +08:00
唐小鸭 d457378c02 feat(scanner): decouple bucket walkers from disk count (#8539)
* fix(scanner): treat replication status writebacks as status-class hits

A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).

The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.

No persisted format, protocol, admin API or configuration changes.

This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.

Refs #8375, #8494, rustfs/backlog#2816.

* fix(scanner): refresh replication statistics during maintenance

Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): avoid expanding the legacy S3 DTO surface

Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(scanner): keep raw enumeration bookkeeping linear per entry

The page writer inserted into its building page with a push-and-sort on
every entry and committed each page through a clone. Insert in sorted
position and move the committed page into the index instead.

Add a regression test that records 20,000 entries in one prefix and pins
the cost to one hash per entry and well under a second, so the quadratic
re-normalization removed by #8114 (rustfs#8113) cannot return once pacing
builds on this walker.

* feat(scanner): decouple bucket walkers from disk count

Add the scanner key walkers_per_disk (RUSTFS_SCANNER_WALKERS_PER_DISK,
1 through 16, default 1). Each local coordinator disk of a set spawns that
many bucket walkers; remote disks keep one. The walker count is what
max_concurrent_disk_scans caps and the disk count stays the floor.

The walkers of one disk share its scanner admission through
LocalDiskAdmissionShare: the first walker to start a bucket takes the
per-disk permit and the last to finish releases it, so a peer's remote
request still sees one scan session per physical disk. Without the share
the second walker was rejected as busy and retired.

The disk_concurrency_budget log line reports walkers_per_disk, walkers and
the reason for the effective limit. The default stays 1 because no local
disk reports rotational media today; the design keeps a higher flash
default as the intent.

* docs(scanner): document walkers_per_disk in the Helm chart and configuration manuals

RUSTFS_SCANNER_WALKERS_PER_DISK was only described in the scanner runtime
controls table. Add it to the Helm chart (values, configmap rendering,
README table, template test), the configuration crate's environment
reference with operator guidance, and a short section in the operations
manual on when to raise it and how to confirm it took effect.

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-16 11:15:45 +08:00
cxymds e76cb67e93 fix(lifecycle): read exact object versions after writes (#8601)
* fix(lifecycle): read exact object versions after writes

* test(lifecycle): respect ECStore dependency boundaries
2026-10-16 11:09:55 +08:00
Hauser 8c8c74cbc3 fix(ecstore): unify bounded delete recovery across upgrades (#8594)
* fix(ecstore): complete delete recovery across upgrades and restarts

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): expect typed quorum error after delete rollback

Assert the exact InsufficientWriteQuorum bucket and object instead of the removed Io wrapper, retaining the rollback and cleanup ownership checks.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-16 11:09:50 +08:00
rustfs-cla-assistant[bot] 7e5d32eecf Merge remote-tracking branch 'origin/main' into gateway/integration 2026-10-16 02:09:53 +00:00
Hauser 9d85d21ee6 chore(deps): refresh workspace dependencies (#8597)
* chore(deps): refresh workspace dependencies

Run cargo update and cargo upgrade twice from origin/main.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(deps): pin Pulsar 6.9.0 until recovery is bounded

Keep Pulsar on 6.9.0 after reproducing background recovery and lookup activity after delivery timeout and target destruction in 6.9.1. Use an exact requirement to prevent a later cargo update from selecting the affected release.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-16 09:54:58 +08:00
overtrue 72da0e21c4 chore: merge main into gateway/integration
Keep the migrated compatibility adapters in their active modules while bringing in upstream fixes. Preserve native publication history on the new create, rename and borrowed-open paths.

Validation:
- `cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(disk::os::native_creation_history_tests::) | test(disk::local::test::borrowed_open_cancellation_retains_started_history_before_execution) | test(disk::local::test::owned_direct_child_open_attempt_consumes_native_history)' --no-fail-fast`: 9/9 passed after restoring all 12 mutation cases at cea78f2590645eb0d303cd7cf56f4dd0d9c00e87; these source blobs are unchanged in this merge.
- `cargo nextest run --locked -p rustfs-s3select-query --lib -E 'test(dispatcher::json_document_schema::tests::)' --no-fail-fast`: 5/5 passed at 5421ba682d1329cc7ebeaac7c6a863da2cea1996, with the same tree.
- `cargo fmt --all --check`: passed.
- `./scripts/check_test_count_ratchet.sh --update`: 52 crates, 22007 tests, 165 ignored.
- Profile baselines use seven actual Darwin listings and seven actual Linux listings.
- Earlier conflict snapshot f5e0cdf0c58aed1ebe53b9243cf52f94832ab049: `cargo nextest run --locked -p rustfs-ecstore -p rustfs --lib -E 'test(disk::os::) | test(bucket::metadata_sys::) | test(server::legacy_compat::) | test(server::layer::tests::request_logging_watchdog) | test(server::http::tests::conditional_object_http::) | test(storage::helper::tests::operation_helper)' --no-fail-fast` passed 218/218 before the borrowed-open fix. This is historical evidence.
2026-10-16 09:49:08 +08:00
Hauser 4e9b3818eb test(s3-select): fix schema reference field ordering (#8599)
Enable preserve_order for the Arrow JSON schema reference in tests and verify first-seen field ordering across nested records.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-16 09:12:37 +08:00
overtrue f0b584d61a fix(ci): stop PRs for sync conflicts 2026-10-16 08:06:27 +08:00
Chris 78a0cf321c refactor(replication): move the replication config onto gateway types (#8443)
* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests
2026-10-16 06:19:32 +08:00
Chris d0fec44808 test(e2e): send the quota batch delete in verbose mode (#8589)
Since #8535 DeleteObjects honours Quiet=true the way S3 does and omits the
successful entries from the response, so the quota batch-delete test has
been asserting two Deleted entries against an empty list on every main run.
The test only needs the batch to succeed and free quota; make it verbose so
the existing assertion holds and check that no entry failed.
2026-10-16 06:07:29 +08:00
唐小鸭 b4f58b81cf test(e2e): gate the rc.5 baseline restarts on drive re-admission (#8595)
* test(e2e): gate the rc.5 baseline restarts on drive re-admission

The mixed-version rolling upgrade failed three times on unrelated branches
with `MultipartAcknowledgementDeadline { node: 2, key: "one-current-node/..." }`.
The baseline kills the next rc.5 node about 10 ms after the previous one
reports ListBuckets, while that node is still replaying its startup traffic.
Two failed operations drop the peer's drive to Offline on the pinned release,
and only the 2 s recovery probe brings it back three successes later. In both
failing runs node 2 held node 3's drive as `faulty` when the multipart load
started three seconds later; it then committed every upload at three of four
drives with a heal per object, and on a two-vCPU runner one upload exceeded
the 30 s acknowledgement budget. The green run on the same branch only
reached `suspect`. An all-rc.5 cluster reproduces the same degradation after
the same restart sequence and an all-current cluster shows none of it.

Write a slash-free probe object before the first restart and, after each
baseline restart, block until every node reports all four drives of that
object as `ok` through a dry-run object heal. `storageinfo` and `info` only
carry each node's local drives; the object heal runs on the receiving node and
classifies a short-circuited remote drive as `faulty`. The gate stays in the
all-rc.5 baseline: admin heal first probes every peer's heal-control
capability, and a mixed-version cluster fails that probe on both binaries.

Record every node's lock table in `mixed-multipart-failure.json`; the stalled
writer is a previous-release node whose table the current node cannot show.
The 30 s budget, layout, load and assertions are unchanged.

* test(e2e): probe drive re-admission with a parity-0 write per node

The dry-run heal gate observed only node 0: the pinned rc.5 admin heal
forwards start and status requests to the heal-control coordinator whatever
node receives them, so the loop checked node 0's drive handles four times
and never the restarted peers' handles.

Run the cluster with `RUSTFS_STORAGE_CLASS_RRS=EC:0` and probe each node
with a reduced-redundancy PUT instead. A parity-0 write on the four-drive
set needs all four drives (`write_quorum = data_drives`), so the PUT fails
with 503 `ServiceUnavailable` until the serving node writes through every
one of its own drive handles. The probe is plain S3 and executes on the
receiving node on both binaries, so the gate now also runs before each
mixed-version load and after the last upgrade. Standard-class objects keep
the default layout.

`parity_zero_write_needs_every_drive` pins the premise on the workspace
build: with one node down a standard write still commits, the
reduced-redundancy write fails with 503, and the gate converges once the
node returns. The diagnostic snapshot test now checks the per-node lock
probe array.
2026-10-16 05:03:31 +08:00
cxymds 7682a12b58 fix(rebalance): wait for readiness and admit peer workers (#8570)
* fix(rebalance): avoid contention when peers join the same run

* fix(rebalance): wait for fleet proof during initialization

* test(rebalance): check contextualized competing run errors

* fix(rebalance): retain typed worker admission lock errors

* test(rebalance): retry the init readiness 503 and pin the retry error shape (#8586)

The e2e harness only retried the pre-fix 500 carrying the fleet-proof
marker; the init stage now answers 503 + Retry-After after its own
readiness budget, so start_rebalance treats that as the same wait. The
missing-proof regression also asserts the typed error shape that the admin
handler's retry predicate depends on, not only the rendered message.

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-15 17:12:26 +00:00
cxymds 65f30bd5c1 fix(multipart): retain commit guards across request cancellation (#8571)
* fix(multipart): retain commit guards across request cancellation

* fix(multipart): preserve commit task error context

* fix(multipart): reclaim verified orphan upload staging (#8572)

* fix(multipart): reclaim verified orphan upload staging

* fix(multipart): share orphan guards across matching lock domains

* fix(multipart): preserve orphan cleanup task error context

* fix(ecstore): box guarded delete transition identity

* fix(multipart): keep NoSuchUpload when orphan recovery cannot prove absence (#8587)

Recovery is a side path of an idempotent Abort. An offline disk, lock
contention, partially overlapping lock domains or unreadable bucket
metadata all stop the staging-absence proof before any deletion; those
used to propagate as 503/500, where the base answered NoSuchUpload. They
now keep that answer and delete nothing; only failures inside the
deletion task remain request failures, because a retry can finish them.

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-15 17:05:48 +00:00
Hauser 7462c25319 feat(diagnostics): port read attribution to main (#8579)
* feat(diagnostics): attribute storage reads to request origins

Port the diagnostics instrumentation from its experiment base onto main
without carrying the base branch's unrelated changes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(diagnostics): consolidate read context scopes

Keep caller and entry provenance in one task-local context, combine the object-read scope, and classify source paths without allocating.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-16 01:02:03 +08:00
Hauser 1c663b8207 fix(rpc): clarify stale epoch rejection diagnostics (#8585)
Read the connection peer address using the stored extension type and distinguish retryable stale epochs from terminal signature errors. Cover diagnostics and authenticated ReadAll recovery without changing replay protection.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-16 01:01:45 +08:00
cxymds cb2dfec17e fix(ecstore): merge migrated copies across inline layouts (#8588) 2026-10-16 01:01:10 +08:00
Hauser c1f906d128 fix(ecstore): quiet routine admin info timing logs (#8584)
Replace unconditional stage warnings with one trace-gated, monotonic
completion summary. Preserve topology and RPC failure diagnostics.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-16 00:26:59 +08:00
唐小鸭 b08331335a test(replication): cover SSE-S3 replicas on ETag-computing targets (#8583)
The outbound target matrix only modeled targets that adopt the forwarded
source ETag, so the post-PUT replica ETag audit was never exercised against
a target that computes its own ETag (AWS S3 and most compatible stores),
and no matrix shape was encrypted. That gap let the SSE-S3 "replica etag
mismatch" regression fixed in #8573 ship unnoticed.

Add an ignore_source_etags mode to the fake S3 target, a ComputesOwnEtags
matrix row that uses it, and an SseS3 object shape (the source runs with the
local KMS). The shape asserts its source ETag is not the plaintext MD5 so
the cell cannot pass vacuously. Against a build without #8573 the new row
fails exactly computes-own-etags/SseS3 with "replica etag mismatch"; with it
the whole matrix passes.

Also correct the #8573 docs and comment that called the replication PUT
checksum the integrity check for encrypted sources: managed-SSE sources
forward no checksum, so their replicas have no post-PUT content audit.
2026-10-16 00:07:54 +08:00
Hauser aae5cc4919 fix(scanner): persist partial progress after budget cancellation (#8578)
* fix(scanner): persist partial progress after budget cancellation

Separate progress-only cycles from observational and authoritative usage
updates. Keep peer movement leases alive under the parent cycle context
for a bounded drain after the scan budget stops the walk, while preserving
the existing strict publication fences for emitted usage updates.

Skip root rename fences and the usage persister when no update was emitted.
Cover real lease admission, cancellation, bounded retries, and durable
partial-cycle advancement without usage publication or dirty acknowledgement.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(scanner): consolidate publication policy and ownership

Classify emitted updates directly in the publication plan constructor and
exercise that single policy from the namespace-admission tests.

Share the parent cancellation state for read-only progress proof checks
and move the one-use root fence into the persister, avoiding a child token
allocation and a JSON string copy without changing lease or commit rules.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-15 23:27:26 +08:00
Chris 69e987b4de fix(ecstore): box the transition payload of CurrentDeleteGeneration (#8581)
`cargo clippy --all-targets -- -D warnings` fails on main since #8535:
`CurrentDeleteGeneration::Present` carries a `CurrentDeleteTransition`
(three Strings, an Option<String>, a 32-byte identity and two small
fields, ~211 bytes) while `Absent` carries nothing, which trips
clippy::large_enum_variant. The workflow only runs clippy on pushes to
main (`if: github.event_name != 'pull_request'`), so the PR CI stayed
green and main's CI has been red since 606d8a3a.

Store the transition behind a Box so the enum is pointer-sized plus the
small fields. The value is transient ("never persisted in object
metadata") and its serde form is unchanged, because serde treats Box<T>
transparently; PartialEq/Clone/Debug derive through it. The two tests
that matched the struct through the Option now bind the Box and check
the same fields in a guard.


Claude-Session: https://claude.ai/code/session_01CezCdjE9AdShFUi9tNk797

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-15 22:48:01 +08:00
Chris 7a32ca2adb fix(replication): skip the replica ETag audit for encrypted sources (#8573)
New encrypted writes store a keyed HMAC fingerprint as the object ETag
(docs/operations/encrypted-etags.md). Replication forwards that ETag on
x-rustfs-source-etag, but a target that computes its own ETag (AWS S3 and
most compatible stores) answers the plaintext PUT with the MD5 of what it
received. Both values are 32 hex characters, so verify_single_part_replica
treated the fingerprint as a comparable MD5 and failed every single-part
SSE-S3 / SSE-KMS replica with "replica etag mismatch"; MRF and resync then
retried the object forever. RustFS and MinIO targets adopt the source ETag
and were unaffected.

Exempt sources whose stored ETag cannot describe the wire bytes: any
managed-SSE or SSE-C source, decided by the existing replication
source-encryption classifier. The checksum header on the replication PUT
remains the integrity check for those objects; plaintext sources keep the
audit unchanged.


Claude-Session: https://claude.ai/code/session_01CezCdjE9AdShFUi9tNk797

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-15 22:01:08 +08:00
Chris d7156290d5 fix(s3): make an unsupported delete participant retryable and explain it (#8574)
A user-bucket delete fans out through the delete-transaction RPC. When a
participant answers Unimplemented (a peer still running a release without
that RPC, or a protocol-version mismatch) the coordinator reports
StorageError::UnsupportedDisk, which had no S3 mapping: clients received a
500 InternalError whose message was the variant's Display text,
"drive does not support O_DIRECT". That text describes the startup drive
probe and sends an operator mid-upgrade after the wrong cause.

Map UnsupportedDisk to 503 ServiceUnavailable with a constant message that
names the protocol-version cause and the remedy (finish upgrading every
node, then retry). The typed source is retained for diagnostics and no
Retry-After is added, since retrying the same node does not help until the
cluster upgrade completes. No request path other than delete recovery
produces this variant; the startup drive probe still fails startup as
before.


Claude-Session: https://claude.ai/code/session_01CezCdjE9AdShFUi9tNk797

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-15 21:48:46 +08:00
Henry Guo 6ec1942ba3 fix(data-usage): retain complete totals when a set is missing (#8558) 2026-10-15 19:09:06 +08:00
Ramakrishna Chilaka 606d8a3a8a fix(scanner): clear transient cycle recovery on follower lock contention (#8566) 2026-10-15 18:36:48 +08:00
Hauser 2c2c1133d4 fix(scanner): use observational leases for incomplete snapshots (#8561)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-15 15:49:38 +08:00
Hauser 8a96fedfdd fix(ecstore): retain pool metadata repair ownership (#8552)
* fix(ecstore): retain system metadata repair ownership

Return system metadata convergence failures to the fenced writer after
committed cleanup instead of submitting an unexecutable MRF handoff.
Drain system rename tails before acknowledging and preserve ordinary
incarnation-bound object repair.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): separate PUT commit and completion outcomes

Keep committed metadata writes on their retained cleanup owner when the
writer must handle repair. Propagate the completion error at the API
boundary without a duplicate temporary-data deletion or worker wait.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): scope repair completion to pool metadata owners

Keep ordinary system metadata quorum-compatible and explicitly select
owner-observed repair completion for fenced pool metadata CAS writes.
Seed format-heal metadata before injecting the missing-format fault.
Preserve degraded bucket recreation and test all completion modes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 21:02:41 -07:00
cxymds ce00d8f7e5 feat(s3): support conditional bulk deletes (#8535)
* feat(s3): support conditional bulk deletes

* fix(s3): report null marker identity in bulk delete responses

* fix(s3): isolate conditional bulk delete actions and tier guards

* fix(tier): preserve backend identity validation diagnostics

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-14 20:20:14 -07:00
Chris cbd53974f9 test(e2e): wait for heal control before deep heal after restart (#8554) 2026-10-15 07:07:12 +08:00
alpbeysir 65ea7db6a9 feat(iam): support wildcard in IAM policy resources (#8555)
Allow wildcard in IAM policy
2026-10-14 15:58:27 -07:00
Hauser bdc00c8d0d fix(s3): align raw disk-full failures with capacity refusals (#8553)
Map DiskFull through the existing StorageFull S3 response contract instead
of falling back to InternalError. Preserve the typed cause and keep lock
retry advice separate from capacity failures.

Reuse the PUT storage-failure fixture to cover capacity refusals and the
existing read-quorum stages without adding a runtime normalization layer.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 10:16:39 -07:00
Hauser 84f7ab1f5c chore(deps): refresh Rustls and TAR parser test crates (#8547)
* chore(deps): update Rustls patch release

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): update TAR parser test crates

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): avoid collecting requests in test

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
2026-10-14 23:23:57 +08:00
Hauser 8b75145a29 fix(s3): preserve catalog lock failures at the error boundary (#8550)
Retain typed retryable catalog lock failures and route S3 publication
errors through the canonical storage conversion for bounded metrics,
public-message redaction, and existing retry advice.

Borrow CAS results for metrics instead of cloning opaque error sources.
Cover authorization under a real publication writer, HTTP serialization,
source preservation, recovery, and established catalog error categories.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 23:16:40 +08:00
唐小鸭 c9f88cff66 fix(scanner): treat replication status writebacks as status-class hints (#8509)
* fix(scanner): treat replication status writebacks as status-class hits

A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).

The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.

No persisted format, protocol, admin API or configuration changes.

This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.

Refs #8375, #8494, rustfs/backlog#2816.

* fix(scanner): refresh replication statistics during maintenance

Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): avoid expanding the legacy S3 DTO surface

Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): seed the ordinary-timeout peer fixture under the default drive deadline

The one-second ENV_DRIVE_MAX_TIMEOUT_DURATION in
target_ordinary_timeout_keeps_its_physical_namespace_owner wrapped the
whole test, so the fixture's write_metadata and the final read_version
were bounded by it too. On a loaded CI runner the seed took longer than a
second and the test failed with "seed real metadata: Timeout" before the
paused rename under test ran. Only the rename now runs under the
one-second deadline; the fixture and the tail read use the default.

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-14 23:10:57 +08:00
Chris 9408b90f13 chore(release): prepare 1.0.2 (#8549)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-14 22:12:55 +08:00
Hauser 08ac1ec218 fix(s3): advise retry after typed lock timeouts (#8546)
Add a one-second Retry-After hint at the canonical S3 error conversion
only for ServiceUnavailable responses retaining a typed lock timeout.
Keep existing error codes, source chains, and lock budgets unchanged.

Cover actual HTTP/XML serialization, nested causes, private-resource
redaction, and unrelated disk/quorum/text failures. Document the response
contract and the limits of log-count and five-second timeout attribution.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 20:52:14 +08:00
cxymds 388d1f17e6 fix(heal): safely recover missing bucket directories by default (#8517)
* fix(heal): safely recover missing bucket directories by default

* fix(heal): allow bound admin recovery before namespace checks
2026-10-14 03:57:09 -07:00
Henry Guo 22c093e66b feat(table-catalog): add opt-in periodic maintenance queueing (#8537) 2026-10-14 03:55:36 -07:00
Chris 24ab34a5ee fix(ci): restore main lint and test reliability (#8543)
* fix(ci): remove needless returns from Linux disk helpers

* fix(test): verify encrypted purge replicas by version body

* fix(test): expire namespace timeout after fixture preparation

* fix(ci): remove remaining needless collection and borrows

* fix(test): wait for active admin heal at interruption checkpoint
2026-10-14 18:28:10 +08:00
Hauser 9068d83400 fix(heal): unblock automatic replacement recovery (#8533)
* fix(heal): unblock automatic replacement recovery

Repair bucket publication directories before adopting survivor delete proofs.
Keep explicit empty-volume removal after proof settlement and preserve the
admitted bucket incarnation for administrator replay.

Discover unfinished intents across every local set and resume the durable
owner with its exact target scope before admitting new sibling replacements.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): route scanner fixtures through the storage boundary

Expose the existing disk-layout bootstrap symbols as test-only aliases in
heal::storage_api and use that boundary from scanner recovery regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(heal): index replacement discovery within each scan

Keep a single endpoint-keyed disk inventory with first-snapshot precedence
and deterministic discovery order. Index local target lookups and move
resume state into generation candidates instead of cloning it.

Retain fresh durable reads and the existing conflict and mutation fences.
Cover duplicate disk handles, endpoint uniqueness and discovery ordering.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): retain committed journals in recovery fixtures

Build one certified survivor delete decision for both replacement and
empty-volume removal tests. Clear target contents without moving its root
inode so Linux disk handles observe the same empty target. Drain PUT tails
and assert the proof remains before the real Heal entry.

Expose the existing recovery state through the ECStore disk API for typed
fixture checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): use server stack for replacement regression

The replacement regression polls the composed Heal future on libtest's
smaller calling-thread stack and aborts before reporting an assertion.
Run it on a bounded 8 MiB test thread and Tokio workers, matching the
debug server's stack budget. Keep all storage and object assertions intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 17:04:36 +08:00
Hauser 20fc897036 chore(deps): refresh crates and use Pyroscope 2.1.2 (#8540)
* chore(deps): refresh crates and use Pyroscope 2.1.2

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): refresh lockfile

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 16:13:57 +08:00
Hauser a86117590c fix(ci): prepare pinned Compose for Connect acceptance (#8531)
* fix(ci): prepare pinned Compose for Connect acceptance

Install the verified Compose CLI through one local action shared by all 11 connected acceptance entry points. Publish only checksum-verified binaries and keep daemon, provenance and timeout gates unchanged.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): use daemon-visible Connect temporary inputs

Match the existing official Connect acceptance TMPDIR contract in all 11 real E2E invocations so daemon bind mounts can read generated inputs. Preserve key modes, read-only mounts and cleanup checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 10:42:35 +08:00
Hauser ceb53fef85 fix(ci): select Python 3.14 for Connect acceptance (#8530)
The shared connected preflight uses uuid.uuid7, which is unavailable in older runner Python environments. Select Python 3.14 with the existing pinned setup action in all eleven acceptance callers.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 09:02:04 +08:00
Nikita Bakun 4e0ff037ec fix(replication): skip version-addressed marker pre-check on targets that mint their own version ids (#8492)
* fix(replication): skip version-addressed marker pre-check on targets that mint their own version ids

* fix(replication): keep bodiless 403 fail-closed and verify resynced markers on targets that mint their own ids

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-13 14:00:54 -07:00
Henry Guo fa0ac13e9b feat(table-catalog): support durable-strong table maintenance (#8523)
* feat(table-catalog): enable durable maintenance control

* fix(table-catalog): revalidate unlocked worker planning

* fix(table-catalog): preserve maintenance state across rename

* fix(table-catalog): fence maintenance reads across rename

* fix(table-catalog): bound scheduler history read fencing

* test(table-catalog): inject loss using the held client lease

* refactor(table-catalog): remove unused metadata log wrapper
2026-10-14 03:54:16 +08:00
Hauser c7848df9a0 fix(build): enforce GNU ABI floor and canonical target metadata (#8524)
* fix(build): pin gnu glibc baseline to 2.31 and report compile target in version metadata

gnu release binaries were built natively on Ubuntu 24.04, embedding
GLIBC_2.38/2.39 symbol references while the RPM/DEB metadata promised
glibc >= 2.31, so Rocky/RHEL 9 (glibc 2.34) could not run them (#8062).
The two linux-gnu legs now build through cargo-zigbuild pinned to the
2.31 baseline (same-arch; zig acts as the linker), with post-build
guards asserting both the embedded version metadata and the highest
referenced GLIBC symbol version, keeping the package declarations
honest.

Version metadata itself now comes from cargo's TARGET cfg instead of
shadow-rs' host stamping, so builds whose host differs from their
target (cross compilation, musl targets on a glibc-hosted toolchain)
report the platform they were compiled for (#8048 follow-up).

The cross toolchain (zig 0.14.x, cargo-zigbuild 0.23.4) is pinned in
the setup action now that it is load-bearing for releases, the build
job gets an explicit matrix-independent name so the connect-*
acceptance workflows keep matching it when matrix fields change, and
the preview-release workflow contract is updated for the TARGET_ARG
build commands.

Fixes #8062
Follow-up on #8048

* fix(build): enforce GNU ABI floor and canonical target metadata

Pin Zig 0.17.0 and cargo-zigbuild, isolate cross-toolchain caches, and
build GNU releases against the declared glibc 2.31 floor. Validate both
release ELF binaries using their loader version requirements.

Generate target metadata once through shadow-rs and check the actual
release version output without adding a parallel metadata facade.

Refs: rustfs/backlog#2824

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(build): reuse GNU release profile across binaries

Switching the CLI back to the default strip profile restarted 419
compilation units after the server build. Keep the profiling build
profile for both binaries, then strip only the CLI artifact.

Retain serial links, server profiling symbols, and post-strip ABI gates.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(build): limit release artifacts to the rustfs server

Build only rustfs in the release workflow and remove the CLI build,
strip, and packaging paths. Preserve GNU profiling symbols and catalog
sidecars, and align strict acceptance archive readers with the new set.

Exercise all six build targets, Linux package checks, and all four
strict acceptance extractors using their actual workflow scripts.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 12:00:40 -07:00
Hauser 4502a1f9b0 fix(ci): repair profiling and compact transition test gates (#8526)
* fix(deps): patch profiling dependency for RUSTSEC-2026-0344

Vendor the published Pyroscope package with an object 0.40 dependency and
lock ruzstd 0.9.1. Preserve runtime sources and profiling features, add
bounded decoder regressions, and run the patched package in audit CI.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>

* fix(deps): correct vendored pyroscope documentation typos

Correct copied comments and changelog spelling for the repository typo
check, and describe these comment-only changes in the vendor provenance.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>

* fix(ecstore): refresh compact transition proof per test case

Install the short-lived homogeneous fleet proof immediately before each
kill-point transaction. Earlier cancellation and recovery work must not
age the next case's writer capability fixture.
Report an early transition result instead of waiting for an unreachable
kill point. Preserve cancellation, remote-owner and payload assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>

---------

Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 11:59:59 -07:00
Hauser 93d51d172f fix(server): cancel request watchdogs with response futures (#8528)
Tie watchdog cancellation to response future ownership before the first
poll, and prefer cancellation when the watchdog deadline is also ready.
Preserve completion logging and genuine slow-request warnings.

Add paused-clock regression coverage for dropped, completed, panicking,
and still-pending request futures.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 11:19:22 -07:00
Hauser b80590f1a7 perf(heal): index replacement history reconciliation (#8527)
refactor(heal): index replacement generation reconciliation

Reconcile owned records and their definitive sources in one pass. Group claims by target slot and use a shared-anchor fast path while retaining pairwise ambiguity checks.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 11:06:21 -07:00
Hauser 025ed4c6d1 fix(heal): preserve retained replacement histories (#8525)
Keep definitive local histories while rejecting ambiguous overlapping peer records.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-14 00:36:33 +08:00
hector 3b428bffef fix(heal): merge disjoint replacement generations in cluster status (#8521) 2026-10-13 08:18:58 -07:00
Hauser da52303f2a fix(server): restore socket peer fallback for client rate limits (#8519)
Read the optional socket peer extension supplied by external ingress so
API and Console client limits remain effective without trusted proxies.
Keep validated client precedence and the existing missing-peer policy.

Exercise the full external stack across HTTP request versions, direct
IPv4/IPv6 peers, forged headers, and distinct trusted proxy clients.
Align API and Console fixtures with the production extension contract.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 20:32:27 +08:00
Chris a79c6072f1 fix: suppress clippy too_many_arguments on rename_data_commit_from_observations (#8515)
fix(ecstore): suppress clippy too_many_arguments on rename_data_commit_from_observations

The function takes 9 slice parameters for the renamedata commit phase.
Annotate with #[allow(clippy::too_many_arguments)] to match the existing
codebase pattern for internal multi-slice commit helpers.
2026-10-13 20:31:23 +08:00
Hauser ab512768d5 fix(ecstore): group rename data commit observations (#8516)
Group disk observations into a borrowed struct to keep the commit helper
within Clippy's argument limit without changing quorum snapshot logic.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 20:05:36 +08:00
cxymds adf17bbfba fix(heal): reject incomplete administrator bucket discovery (#8512) 2026-10-13 04:12:28 -07:00
GatewayJ fb5c002a30 refactor(iam): encapsulate federated authorization models (#8514) 2026-10-13 03:59:29 -07:00
Hauser 8a94c64934 fix(audit): preserve request identity and verified client address (#8513)
Capture the authenticated request access key when building the audit entry
and preserve it through completion instead of replacing it with root
credentials. Resolve remotehost from validated client context or the
socket peer, and ignore untrusted forwarding headers.

Cover request identities, completion outcomes, client address precedence,
and audit serialization without credential secrets.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 18:56:44 +08:00
Hauser 9ca0e9b717 fix(obs): isolate bucket usage dashboards by cluster (#8510)
Keep same-name buckets in independent clusters separate while deduplicating
replicated usage observations. Document the existing scanner-backed exporter
and cover its gauge contract and cluster-scoped queries.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 03:00:44 -07:00
Henry Guo bd83bd3f6b fix(table-catalog): allow commits after object-backed rename (#8508) 2026-10-13 17:12:00 +08:00
Hauser 34803e8165 perf(ecstore): isolate cleanup and preserve metadata authority (#8289)
* chore: pin s3s to upstream git revision

Use the upstream s3s git repository at 50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6 for both s3s and s3s-sigv4.

Allow the official s3s git source in cargo-deny so source checks continue to pass.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): instrument put object hot path

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): split cleanup worker wait metrics

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* upgrade version

* chore(metrics): add PUT post-store stage probes

Add lightweight PUT post-store stage probes for issue 2615 validation.

* fix cargo deny check warn

* upgrade version

* perf(ecstore): bound and isolate cleanup execution (#8151)

* perf(ecstore): bound and isolate cleanup execution

* fix(ecstore): preserve cleanup cancellation and lease release

* feat(ecstore): default isolated cleanup to CPUs 1-2

* test(ecstore): decouple budget tests from CPU affinity

* upgrade version

* perf(ecstore): reuse pinned directory handles for object I/O

* feat(observability): add sampled local GetObject stage summaries

Add an opt-in GetObject stage aggregation path that works without the
OTLP metrics export. When RUSTFS_OBS_GET_STAGE_LOCAL_SUMMARY_ENABLED is
set, io-metrics keeps per-(path, stage, object class, size bucket)
sampled counts and duration totals in a bounded in-process map, and a
startup task flushes them periodically to
/var/log/rustfs/get-stage-summary.log, tunable via the sample rate and
interval env keys.

* build(deps): bump workspace dependency releases

Patch releases: async-compression 0.4.49, tokio-rustls 0.26.6,
xxhash-rust 0.8.19, suppaftp 12.1.1, russh-sftp 3.0.1. Minor release:
hotpath 0.27.0.

* fix(ecstore): fence distributed metadata cache mutations

Add authenticated cluster-wide mutation fences for the opt-in DistErasure GetObject metadata cache. Guard recursive prefix and batch deletes, fail closed on configuration mismatch or uncertain quorum, and cover replay, restart, peer loss, and mixed-configuration behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(metrics): consolidate PUT observers on improve/base (#8039)

* perf(object): instrument storage hot paths

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(metrics): consolidate PUT stage observation

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>

* build(deps): bump workspace dependency releases

* ci: allowlist rustix OFlags WRONLY in typos config

* refactor(storage): route ecstore symbols through storage_api boundaries

* perf(ecstore): cache quorum-selected early-stop metadata (#8308)

Publish eligible latest-version GET metadata after a validated read quorum
when early-stop cancels the remaining fanout. Keep metadata errors, selection
quorum, online read quorum, and mutation-generation checks as publication gates.
Add slow-peer repeat-hit and concurrent mutation-fence regression tests.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* chore: merge latest main into improve/base (#8332)

* refactor(storage): bind rename rollback to its commit context (#8324)

* refactor(storage): separate metadata scheduling from quorum decisions (#8327)

* test(storage): verify rejected rollback inspection on real disks (#8331)

* test(storage): make commit fault checkpoints independent (#8326)

* test(storage): make commit fault checkpoints independent

* test(storage): simplify commit checkpoint lookup

* fix(listing): count logical prefixes before pagination (#8328)

* fix(listing): count logical prefixes before pagination

* test(listing): normalize the fixed corpus seed

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): borrow metadata cache lookup key (#8333)

Avoid request-scoped Arc<str> allocations on eligible cache lookups while preserving full key identity.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* chore: sync improve/base with latest main (#8334)

* refactor(storage): bind rename rollback to its commit context (#8324)

* refactor(storage): separate metadata scheduling from quorum decisions (#8327)

* test(storage): verify rejected rollback inspection on real disks (#8331)

* test(storage): make commit fault checkpoints independent (#8326)

* test(storage): make commit fault checkpoints independent

* test(storage): simplify commit checkpoint lookup

* fix(listing): count logical prefixes before pagination (#8328)

* fix(listing): count logical prefixes before pagination

* test(listing): normalize the fixed corpus seed

* fix(scanner): isolate and coalesce dirty usage journals (#8325)

* fix(scanner): isolate and coalesce dirty usage journals

Bind dirty replay records to a stable node owner so peers cannot overwrite
or delete another node's pending journal. Replace the unbounded command
queue with bounded coalesced state and revision-aware background saves.
Keep legacy records isolated and enforce replay byte limits with conservative
whole-bucket or unverified fallback. Add publication-stage diagnostics and
storage regressions for ownership, clearing, retry, and restart behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): collapse confirmed journal clear condition

Use a let-chain to retain the generation fence before removing a confirmed
bucket and updating its scope byte accounting. Satisfy collapsible_if
without changing journal clearing behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(obs): distinguish metrics by process identity (#8343)

Attach process.pid to the OpenTelemetry resource and promote it for Prometheus
so counter series from restarted processes remain distinct. Preserve the stable
RustFS node and service instance attributes.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* merge: sync origin/main into improve/base (#8345)

* refactor(storage): bind rename rollback to its commit context (#8324)

* refactor(storage): separate metadata scheduling from quorum decisions (#8327)

* test(storage): verify rejected rollback inspection on real disks (#8331)

* test(storage): make commit fault checkpoints independent (#8326)

* test(storage): make commit fault checkpoints independent

* test(storage): simplify commit checkpoint lookup

* fix(listing): count logical prefixes before pagination (#8328)

* fix(listing): count logical prefixes before pagination

* test(listing): normalize the fixed corpus seed

* fix(scanner): isolate and coalesce dirty usage journals (#8325)

* fix(scanner): isolate and coalesce dirty usage journals

Bind dirty replay records to a stable node owner so peers cannot overwrite
or delete another node's pending journal. Replace the unbounded command
queue with bounded coalesced state and revision-aware background saves.
Keep legacy records isolated and enforce replay byte limits with conservative
whole-bucket or unverified fallback. Add publication-stage diagnostics and
storage regressions for ownership, clearing, retry, and restart behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): collapse confirmed journal clear condition

Use a let-chain to retain the generation fence before removing a confirmed
bucket and updating its scope byte accounting. Satisfy collapsible_if
without changing journal clearing behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(storage): retain typed owners across borrowed writes (#8329)

* fix(storage): retain typed owners across borrowed writes

* fix(storage): preserve Copy write completion policy

* refactor(storage): remove redundant write owner references

* fix(ecstore): hide rollback backup residue in prefix listings

* test(connect): isolate short HTTP fixtures in nextest

* fix(protocols): enforce S3 policies for file protocol operations (#8262)

* test(ecstore): enforce main core regression gates (#8338)

* test(ecstore): enforce main core regression gates

* test(ecstore): pin external shards and completion regressions

* fix(test): restore authenticated cluster fault scenarios (#8340)

* fix(test): restore authenticated cluster fault scenarios

* fix(test): use the E2E storage API boundary

* ci: coerce the pool timeout input to a number

timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes
is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to
the string '240' and the runner rejected the template: 'An error occurred
when attempting to determine the step timeout ... Unexpected value 240'
(chain run 36505625402 job at 02:25:52, standalone run 36393453114 at
07:46:37). Both pool runs since #8155 merged have therefore executed
without the intended step budget. Wrap the expression in fromJSON and
update the policy test that pinned the broken literal.

* ci: coerce the pool timeout input to a number

timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes
is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to
the string '240' and the runner rejected the template: 'An error occurred
when attempting to determine the step timeout ... Unexpected value 240'
(chain run 36505625402 job at 02:25:52, standalone run 36393453114 at
07:46:37). Both pool runs since #8155 merged have therefore executed
without the intended step budget. Wrap the expression in fromJSON and
update the policy test that pinned the broken literal.

* fix(ecstore): own listing workers through cancellation (#8342)

Consolidate the three page collectors, preserve completed worker failures, and abort nested merges when their owner is dropped.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): route commit sync through guarded helper

Use the Linux-capable guarded directory sync entry point for both commit sync sites. This keeps group commit and file-sync admission behavior while ensuring the already-open commit directory remains the sync target.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): expose rename directory guard in production

The commit path now acquires the directory guard in production builds, not only tests, so destination metadata reads and durable sync share the guarded context.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>

* fix(ecstore): remove unsafe metadata cache admission paths

Keep distributed GET metadata on quorum reads and quota admission on authoritative storage. Preserve local cache invalidation, remove the experimental RPC surface, and gate diagnostic work.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): cover cold degraded mutation admission

Check first PUT and DELETE with one peer absent and no SDK retries, then verify that losing write quorum still rejects writes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): retain guarded Linux destination metadata reads

Use the existing commit directory guard when reading old xl.meta on Linux. Cover symlinked destination metadata for inline and non-inline renames without changing the split commit structure.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(test): remove retired cache RPC import

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* merge: sync latest main into improve/base (#8349)

* test: allow cleanup fixture CI headroom (#8336)

* fix(s3): encode multipart listing response keys (#8347)

---------

Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* merge: sync main auth and scanner updates (#8352)

* test: allow cleanup fixture CI headroom (#8336)

* fix(s3): encode multipart listing response keys (#8347)

* perf(scanner): reduce scan and checkpoint peak allocations (#8346)

* perf(scanner): reduce scan and checkpoint peak allocations

* test(scanner): isolate partial commit recovery scenarios

* fix(auth): return ExpiredToken for retained STS credentials (#8344)

* fix(auth): return ExpiredToken for retained STS credentials

Validate session identity independently of credential expiry in both IAM
rejection and claims validation, and preserve typed S3 errors at the caller.
Keep token binding, disabled-account precedence, and service-account behavior.
Add exact expiry, IAM, and HTTP/XML regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): synchronize the metadata slowtail hedge regression

Park the slow metadata task at a test-only fault checkpoint and release
healthy reads after the fault is active. Assert the spare response and
cancelled-task drain without imposing a 300 ms disk/scheduler deadline.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ilm): synchronize copy compensation transition stages

Observe automatic compensation reaching the mock tier PUT checkpoint before
waiting for the final metadata commit. Assert forced enqueue failure,
automatic backfill, one completed tier upload, and payload integrity.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reject overlapping cleanup schedulers

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reject overlapping cleanup schedulers

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(ecstore): explain cleanup scheduler conflict

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(metrics): separate GET request and timeout outcomes (#8354)

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* docs: refresh security advisory lessons (#8350)

* fix(heal): bound durable MRF rescheduling (#8351)

Reuse stable live-owner proof anchors to avoid duplicate in-flight admission. Pace unverified completions and unavailable attempts with bounded jitter while retaining checkpoints, exact proofs, and generation isolation.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(rustfs): route recovery errors through S3 gateway

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): update workspace dependencies

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(io-metrics): use is_multiple_of for sampling

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): satisfy strict Clippy pagination checks

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): avoid cloning Copy purge identity

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* add ignore .mimosa

* fix(zip): adapt archive readers to zip 9 APIs

Upgrade zip and its required dependencies alongside fallible archive-name handling in log ingestion, report bundles, bucket metadata import, and archive tests. Preserve existing skip and error propagation behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reuse quorum metadata for body GETs

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): align storage test stacks and peer reset classification

Apply the existing bounded ECStore stack setup to the affected cross-crate storage fixtures. Validate conditional-only metadata publication separately from full body-layout cache reuse. Classify typed h2 shutdown resets as unreachable while retaining application, authentication, cancellation, and deadline distinctions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): update workspace dependencies

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve omitted-disk GET recovery

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore: integrate published workspace dependency updates

Preserve the concurrent dependency refresh and retain the typed h2 reset classification and bounded storage-test stack setup.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): preserve snapshot size budget after fallback move

* fix(test): use one barrier for compensation COPY

Let the forced-enqueue helper own the PUT barrier and compensation assertions. Waiting on an outer barrier overwritten by the helper always timed out after the upload had completed.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(ecstore): apply current lint fixes

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): update sha2 to 0.11.1

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: GatewayJ <835269233@qq.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: cxymds <cxymds@gmail.com>
2026-10-12 22:07:50 -07:00
cxymds 9af2f7c04e fix(ilm): make missing lifecycle expiry targets idempotent (#8504)
fix(ilm): treat missing expiry targets as success
2026-10-12 20:05:21 -07:00
cxymds 9d6f48c325 fix(s3): match AWS duplicate bucket responses (#8503) 2026-10-12 19:25:12 -07:00
Hauser e7e6556f83 fix(scanner): bound superseded waits for regular maintenance (#8502)
Preserve ordinary lifecycle and replication cadence while retaining
usage-only convergence backoff, explicit cycle floors and publication
fencing. Cover repeated snapshots and the real convergence waiting path.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-13 07:04:27 +08:00
Hauser 0a6aeda191 test(e2e): bound scanner witnesses by the reported schedule (#8501)
* test(e2e): bound scanner witnesses by the reported schedule

Require a fresh completed observed walk and account once for planned
superseded waiting, preserving the active-cycle execution budget.
Cover invalid schedules and early wakeup, and refresh exact platform
selection digests without removing existing tests.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): refresh nightly scanner budget membership

Record the three scanner budget regressions in both platform nightly
selection digests, preserving every previously selected test.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 13:06:42 -07:00
overtrue cbe38a120f Merge main into gateway/integration
Merge frozen main acd26afd8b and include
compatibility fixes and measured Darwin/Linux membership corrections
across six files.

Validation on candidate b4c3ab560a:
- All 20 local gates passed.
- Native smoke: 218 passed. Native full: 661 passed.
- Full passed on the same-head failed-only rerun; the initial failure
  remains recorded.
- Gateway remains report-only: 38 passed, 180 failed, under the existing
  exemption.

CI: https://github.com/rustfs/rustfs/actions/runs/38285788334
2026-10-13 03:26:16 +08:00
Hauser 0e0f656ae1 test: isolate license and rebalance cancellation fixtures (#8500)
Preserve canonical claim encoding for the signature tamper test and pause the admin cancellation fixture at its existing run fence instead of durable deletion.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 10:59:46 -07:00
Hauser a57a3ae51d fix(scanner): reuse observations and expose complete usage (#8498)
* fix(scanner): reuse observations and expose complete usage

Allow superseded retries to reach the existing validated cache planner while
bootstrap remains pending. Serve complete observations in server info without
changing quota authority or dirty acknowledgements.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(scanner): route fixture setup through storage boundary

Keep the single-disk scanner fixture on the existing storage_api boundary
instead of importing the ECStore API facade directly.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 09:50:59 -07:00
GatewayJ 5d8e143505 fix(admin): add OIDC policy log fields (#8496) 2026-10-12 08:07:46 -07:00
Hauser d6fe5c30ce test(e2e): borrow URLs in internode RPC helpers (#8497)
Pass existing string slices directly to the client constructor to satisfy strict Clippy without temporary allocations.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 23:02:02 +08:00
Chris ac6cf8a50d fix(test): discover data dirs by listing instead of info.data_dir (#8495)
The conditional_get_304_uses_metadata_without_decoding_damaged_data test
constructed the data directory path from info.data_dir.to_string(), but
the on-disk directory name may not match the UUID string form. Switch to
iterating subdirectories of the object path (matching the
delete_object_part_shards helper) so the test is robust regardless of
the data_dir naming convention.

Verified: cargo nextest run -p rustfs -- conditional_get_304 → PASS
2026-10-12 23:01:37 +08:00
Hauser a7188a16b4 test(storage): make recovery and upgrade checks deterministic (#8491)
* test(upgrade): separate rc.5 data and candidate query checks

Verify historical fixture bytes before upgrading without requiring the
pinned rc.5 reader to satisfy the candidate's cross-part range contract.
Reuse the complete-body oracle and retain all upgraded range, partNumber,
attributes and replication assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): replace timing races with recovery evidence

Prove no_lock batch progress with the existing cleanup barrier while the
caller retains its guard. Check namespace counters and every drive's
metadata absence rather than using a one-second disk I/O deadline.

Keep durable ILM's manual recovery as the sole tier journal consumer and
verify exact migrated input before retaining the original recovery stats,
receipts and remote-version assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 21:17:58 +08:00
Chris 0195732a1f test(ecstore): cover delete cleanup crashes and retries (#8471)
fix(ecstore): preserve retry evidence during delete cleanup

Bind normal cleanup to the coordinator transaction after quorum succeeds, and remove the rollback backup before marked data. Preserve data and markers if backup cleanup fails.

Add deterministic LocalDisk crash and retry regressions plus real four-disk single and batch coordinator coverage. This prevents one post-quorum cleanup window; it does not resolve ambiguous inline or historical residue in #8464.

Co-authored-by: Zhengchao An <overtrue.us@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 21:02:03 +08:00
cxymds 0ef133bbb0 fix: refresh peer metadata without blocking bucket mutations (#8486)
* fix: refresh peer bucket metadata after creation

* fix(s3): release bucket mutation slots before peer reloads

* test(ecstore): distinguish delete lock waits from disk latency
2026-10-12 05:03:16 -07:00
GatewayJ 5228463a1c fix(oidc): recover providers after startup discovery failures (#8475)
* fix(oidc): recover providers after startup discovery failures

* fix(oidc): report recovery failures with throttled diagnostics
2026-10-12 03:40:07 -07:00
Hauser dd90bb7488 fix(s3): short-circuit conditional reads at metadata quorum (#8459)
* fix(s3): evaluate conditional reads before opening object bodies

Reuse locked metadata snapshots for conditional GETs and unify HEAD
preconditions and cache headers. Preserve body-cache hooks and validate
encryption metadata and key authorization before returning bodyless errors.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(s3): unify conditional reads at metadata quorum

Apply scoped metadata conditions to the existing reader pipeline instead
of maintaining parallel prepared-read branches. Preserve quorum checks,
early-stop, coalescing, and cache-hook behavior on condition misses.

Track metadata-only termination through selection and reject body access
or a changed decision. Keep concrete reader futures off request stacks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(s3): strengthen conditional read acceptance evidence

Assert zero shard, decode, cache-fill, and remote-body calls on matching
conditions with live positive controls. Exercise GET and HEAD through
s3s and Hyper with conditional-header and precedence matrices.

Add reproducible paired application-read benchmarks with isolated
metadata controls and strict drift, artifact, and resource gates.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(s3): remove obsolete conditional comparison helper

Keep ETag comparison tests on the canonical read-condition evaluator,
remove unused date-trait imports, and name the HTTP case type.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(s3): validate full conditional etag fields

Reject adjacent tags and malformed trailing list members before accepting a
match. Keep the existing request-scoped metadata read and authorization order,
while removing redundant validation wrappers and unused benchmark overrides.
Replace simulated phase coverage with real 304/412 cache-fill probes and extend
HTTP/1.1 and HTTP/2 ingress regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): cache terminal conditional metadata safely

Preserve quorum-validated conditional metadata when early termination
cancels the remaining fanout. Restrict those entries to terminal GET
readers; body and full-layout consumers still obtain their own metadata.

Serialize cache publication per key so conditional entries cannot replace
complete layouts or renew their TTL. Retain generation fencing and the
final selected-metadata predicate check before any body access.

Cover real four-disk hedging, nonterminal body and range reads, consumer
isolation, generation races, atomic publication and actual cache expiry.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(log-analyzer): align heal writer failure rule with current logs

Match the current structured all-targets failure, retain legacy log support, and reject partial or unrelated failures.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 16:59:21 +08:00
GatewayJ 0018bbf5d7 ci: guard OIDC architecture boundaries (#8447)
* ci: guard OIDC architecture boundaries

* fix(ci): guard OIDC session ownership

* fix(ci): ignore OIDC authorization destructuring

* fix(ci): accept typed authorization destructuring

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-12 16:13:36 +08:00
Hauser 13ee76a4a9 fix(ecstore): validate recovery evidence before mutation (#8487)
Reject causal cycles before source repair and share conservative backup
classification between discovery and inspection. Fence public raw deletion
and derive exact-key removals from successful settlement without an extra
metadata read. Preserve the existing physical phases and strict source checks.

Make outage and renewal fixtures deterministic without weakening their
convergence assertions, and complete strict TLS/RPC lint cleanup.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 01:06:01 -07:00
cppudge 3dc09c9397 test(ecstore): deflake bucket sentinel and dispatch rollback tests (#8488)
* test(ecstore): bound sentinel tests by the lock wait, not recreation IO

Both bucket incarnation sentinel tests gave the bucket recreation that
follows the paused delete (force delete_bucket, make_bucket, put_object
on four fsync'ing disks) 10 s in total. The delete releases the sentinel
milliseconds after the barrier, but on a runner starved for IO the
recreation alone can exceed 10 s:
delete_objects_holds_bucket_incarnation_sentinel_through_commit failed
that way on main in run 37760321449 (job 113259251979).

That bound is not what catches a kept sentinel: delete_bucket waits for
it on its lifecycle write lock, which fails after the lock acquire
timeout, and the recreation's expect reports the lock error. Keep that
check and give the recreation 120 s only as a hang guard.

The recreation starts that lock wait before the barrier is released, so
the wait also has to cover the delete's own commit, and the 5 s default
leaves a starved runner little room for it. Run both tests with
RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=30 and bound the delete, which was
not bounded, by the same 30 s. A kept sentinel still fails the tests,
after 30 s instead of 5 s.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* test(ecstore): keep background recovery out of dispatch rollback tests

dispatch_manifest_detached_workers_are_store_bounded_and_shutdown_cancellable
seeds six aborting dispatch manifests, installs a pause hook and expects
its own recovery passes to park four detached workers on it. Its first
store also ran the background tier-delete journal recovery loop, which
runs a pass every TIER_DELETE_JOURNAL_RECOVERY_INTERVAL (60 s). When the
setup took longer than that, a background pass rolled the manifests
back before the hook was installed, no worker paused, and the 30 s wait
failed. This happened on main on a runner starved for IO, in run
37760321449 (job 113259251979).

Build the first store with background recovery suppressed, as the
neighbouring dispatch tests do. The restarted store keeps it: the second
half of the test checks that it picks up the retained manifests.

dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member
and dispatch_manifest_page_timeout_detaches_one_deduplicated_worker_until_convergence
have the same exposure: each seeds an aborting manifest on a store with
background recovery and only then installs its slow-delete hook. They
have not failed in CI, but with the setup stretched past 60 s they fail
the same way. Suppress background recovery for them too.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

---------

Signed-off-by: Petr Slavyagin <disident47@gmail.com>
2026-10-12 01:02:33 -07:00
Hauser 71c0efa97d fix(scanner): account remote progress and preserve dispatch limits (#8485)
* fix(scanner): account remote progress and preserve dispatch limits

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): isolate remote progress by cycle

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-12 00:29:10 -07:00
Hauser 4166b9503f fix(ecstore): recover interrupted delete transactions (#8468)
* fix(ecstore): recover crash-interrupted delete retries

Retain set-quorum cleanup proof and restore metadata before dropping rollback
markers. Reclaim eligible exact-key residue during explicit delete retries,
with bucket identity, version, retention and namespace fencing guards.

Preserve prefix children and snapshot ownership through batched physical
cleanup, and cover interrupted transactions and failure boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): route recovery test imports through contracts

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve missing-key deletes at quorum

* feat(ecstore): add certified set-wide delete recovery

Atomically prepare deletion journals and exact targets, publish durable set
receipts, and recover through one fenced disk/RPC core used by startup and
scoped administrator healing. Preserve returning generations, ordinary
missing-key quorum semantics, and legacy rollback wire compatibility.

Retain namespace and volume ownership through late physical operations,
order related transactions causally, and keep proof until full convergence.
Remove the superseded local-only recovery path and add storage, protocol,
RPC, durability and isolated Docker regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): reclaim completed multipart delete sidecars

* fix(ecstore): recover multipart sidecars with verified proofs

Recognize native multipart sidecars at the shared recovery boundary and
restore them without treating metadata or proof files as shard payloads.
Validate required proof generations and bind every present live/staged
shard copy to its metadata-bound Merkle proof before restoring metadata.

Cover protected native MPU rollback, split source layouts, missing or
corrupt proofs, and payload corruption with recomputed bitrot checksums.
Add real multipart Docker regressions in both integrity modes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): settle superseded committed version deletes

Allow a validated committed decision to finish its own data cleanup after
a later decision removes current metadata. Prepared copies need an
inspected same-transaction receipt matching immutable local intent before
promotion, and retain their local journal without importing peer hashes.

Reuse the authority matcher for dry-run and settlement. Cover chained
explicit-version deletions, partial receipt publication, mismatch rejection
and no-resurrection, while retaining the no-decision absence protection.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): clarify delete recovery case label

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): restore recovery CI guard coverage

Keep production error sites visible to the ratchet by placing test modules last. Preserve native encoding errors, group preparation identity and ownership, and reduce RPC enum size without changing the wire payload.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(admin): preserve owned OIDC validation errors

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): unify recovery authority and snapshot work

Use one user-bucket legacy settlement boundary and one adoption source predicate for preview and execution. Reuse immutable snapshot digests, consume owned generation metadata, index version membership, and prepare empty witnesses through the canonical journal path.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): preserve delete recovery invariants

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-12 13:47:37 +08:00
cxymds 201b5f62d2 fix(s3): normalize PUT read quorum failures as unavailable (#8484) 2026-10-11 21:55:23 -07:00
Chris 902c97ce05 fix: auto-repair breaking changes from acd26afd8 (#8483) 2026-10-11 20:09:55 -07:00
dependabot[bot] 7ad4fa439e build(deps): bump the dependencies group across 1 directory with 3 updates (#8481)
Bumps the dependencies group with 3 updates in the / directory: [datafusion](https://github.com/apache/datafusion), rust-embed and [hotpath](https://github.com/pawurb/hotpath-rs).


Updates `datafusion` from 55.1.0 to 55.2.0
- [Changelog](https://github.com/apache/datafusion/blob/main/CHANGELOG.md)
- [Commits](https://github.com/apache/datafusion/compare/55.1.0...55.2.0)

Updates `rust-embed` from 8.12.0 to 8.13.0

Updates `hotpath` from 0.28.5 to 0.28.6
- [Release notes](https://github.com/pawurb/hotpath-rs/releases)
- [Changelog](https://github.com/pawurb/hotpath-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pawurb/hotpath-rs/compare/v0.28.5...v0.28.6)

---
updated-dependencies:
- dependency-name: datafusion
  dependency-version: 55.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: hotpath
  dependency-version: 0.28.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: rust-embed
  dependency-version: 8.13.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-11 19:09:19 -07:00
Hauser 1266d4c270 feat(tls): reload and publish external certificate generations (#8467)
* feat(tls): reload external certificate snapshots coherently

Pin projected certificate generations, validate complete TLS material, coordinate server consumers, and invalidate outbound RPC caches coherently. Add externally managed TLS Secret support and rotation guidance.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(tls): publish server certificates through a fenced admin API

Add opt-in server-leaf publication with dedicated IAM actions, actual TLS
transport checks, immutable generations, durable captured-token CAS and
idempotent receipts. Preserve independent trust and client identity sources
and distinguish visible publication state from the full serving chain.

Validate 64 TLS tests, 41 TLS-runtime tests, the focused IAM policy test,
and eight live local API scenarios including least privilege and restart.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(admin): route certificate handler types through S3 facade

Use the existing admin S3 facade to keep direct s3s imports within the footprint ratchet.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(admin): include TLS certificate routes in route matrix

Account for the registered certificate-state GET and certificate publication POST without changing route behavior or weakening matrix assertions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(tls): reconcile startup trust before skipping reloads

Compare the initialized listener bundle with coherently published outbound material, retain startup trust and client identity when sources disappear, and cover independent startup rotations through real reload passes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(tls): reuse pinned generations and prepared listener state

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(tls): preserve startup trust source requirements

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-12 08:53:11 +08:00
Anton acd26afd8b fix(scanner): skip child prefixes removed before they are opened (#8479)
A child prefix can be deleted after its parent listed it and before scan_folder opens it. read_dir then returned NotFound, which propagated through the compacted subtree and dropped the whole top-level child (child_scan_failed), so the cycle never produced a durable usage snapshot under continuous deletes.

Treat NotFound on open like a directory that vanished during iteration for non-root folders. A missing bucket root still reports NotFound so namespace removal handling is unchanged.

Refs #8478

Co-authored-by: Anton <5527364+basic-system@users.noreply.github.com>
2026-10-11 15:25:14 -07:00
Rafael Kansy 5924a55931 fix(obs): honor RUSTFS_OBS_USE_STDOUT with per-signal endpoints (#8476)
use_stdout was forced to true whenever the root RUSTFS_OBS_ENDPOINT was
empty, so a deployment that sets only a per-signal endpoint such as
RUSTFS_OBS_TRACE_ENDPOINT could not turn the stdout exporters off: every
exported span was also printed to stdout as a multi-line text block.

Count a non-empty trace, metric or log endpoint as a configured endpoint.
With no endpoint at all the stdout default is still forced on. Add a test
covering each per-signal endpoint and an empty value.
2026-10-11 13:00:24 -07:00
overtrue db0828aa1c test(ecstore): isolate rollback guard loss fixtures
Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.
2026-10-12 00:42:51 +08:00
GatewayJ 2d2679a710 refactor(oidc): isolate config updates and replication view (#8446) 2026-10-11 08:53:29 -07:00
Chris 257c9f11ae docs: refresh security advisory lessons (#8472) 2026-10-11 22:43:28 +08:00
Hauser 1324e296f8 test(scanner): bound expansion fixture preparation I/O (#8473)
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-11 22:43:08 +08:00
Hauser 38263ba896 fix(log-analyzer): match structured heal writer failures (#8469)
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-11 20:46:44 +08:00
overtrue 3ca889b099 fix(ecstore): box nested bucket rollback state
Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.
2026-10-11 20:32:00 +08:00
overtrue e6112c526f fix(ecstore): retain creation fences through rollback
Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.
2026-10-11 16:16:31 +08:00
Chris c5967686f1 fix(ecstore): bound bucket creation stack usage (#8466) 2026-10-11 00:55:29 -07:00
Chris ba2047b184 ci: reuse workspace test builds for serial ILM checks (#8465) 2026-10-11 13:55:48 +08:00
GatewayJ da42266bfb refactor(iam): organize OIDC runtime modules (#8445) 2026-10-11 13:27:46 +08:00
Chris 367f13f3f9 ci: reduce repeated PR runner work (#8463) 2026-10-10 22:19:25 -07:00
Hauser 0263d5fcb9 fix(storage): preserve heal failures and timeout responses (#8462)
* fix(storage): preserve heal failures and timeout responses

Stop repairs with no writable target before opening source readers. Consolidate target failure aggregation, preserve typed causes, and distinguish all-ignored diagnostics from actual successful nil entries. Map typed disk timeouts to retryable S3 responses without changing quorum decisions.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(heal): unify staging failure cleanup

Route data-stage failures through one local result boundary after temporary
readers and writers are released. Preserve typed report failures, outer
reconstruction errors, partial repair, and the existing commit fences.

Exercise actual source loss after second-part writer admission using the
existing test pause registry. Assert all staged parts are removed, committed
metadata stays unchanged, and restoring the source allows a full-byte retry.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-11 12:41:13 +08:00
Chris 93a64fdca3 fix(dev): restore VS Code debug configurations (#8450) 2026-10-11 12:34:41 +08:00
Chris 859de565ad refactor(rpc): serve internode rpc bodies with the owned Body (#8441)
* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries
2026-10-11 12:34:14 +08:00
Hauser 9ecb00d388 fix(scanner): avoid placeholder retention and resume checkpoints (#8460)
* fix(scanner): bound old-cache lookups and recover checkpoint saves

Keep missing child lookups read-only and preserve the checkpoint receiver
after ordinary persistence errors. Reload on stale CAS or indeterminate
revision refresh, and report confirmed periodic saves separately.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): remove redundant checkpoint error conversion

Return the existing StorageError directly so strict Clippy passes.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs(scanner): clarify checkpoint persistence guarantees

Distinguish a confirmed periodic save from validated restart coverage.
Keep adoption and subtree-skip requirements explicit.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): preserve resumable periodic coverage

Materialize active ancestor accounting in bounded periodic snapshots and
seal completed coverage independently of raw enumeration state. Preserve
durable coverage during early resumed enumeration and exclude unfinished
out-of-order heal work from periodic and cancellation receipts.

Reuse ancestor frame slots to avoid growing child-set copies on descent,
retain compacted accumulator safety, and report checkpoint adoption resets.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): reject covered active ancestors in checkpoints

Check every suspended ancestor alongside the current folder before
certifying completed coverage. An unfinished covered ancestor must not
become a resumable subtree merely because its descendant sorts later.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): retain an incomplete root for missing namespaces

Create the empty bucket root explicitly at the missing-namespace boundary.
Preserve nonempty durable caches and keep namespace results unpublished.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-10 20:54:12 -07:00
cxymds b8889fccc9 fix(ecstore): allow cold-start recovery at EC read quorum (#8458)
* fix(ecstore): load existing formats at EC read quorum

* fix(ecstore): complete startup at EC read quorum
2026-10-11 11:50:24 +08:00
GatewayJ a424d10fa0 refactor(iam): construct OIDC runtime directly (#8436)
* refactor(iam): remove legacy OIDC federation paths

* refactor(iam): construct OIDC runtime directly
2026-10-10 04:01:40 -07:00
Chris 072b57b375 fix: auto-repair breaking changes from 446479791 (#8451)
* fix: repair s3s footprint ratchet violation, clippy warnings, and stack overflow

- Replace s3_error! macro with S3Error::with_message in ecfs.rs to keep
  s3s footprint ratchet at baseline (1586) after retention validation refactor
- Remove redundant clone on Copy type in list_objects.rs (clippy::redundant_clone)
- Remove clone_on_copy calls in access.rs test (clippy::clone_on_copy)
- Use manual thread with 8MB stack for heal_concurrent_delete_test to fix
  stack overflow on macOS (consistent with sibling tests in same file)

* test: align CI assertions with version authorization rules
2026-10-09 17:01:48 -07:00
GatewayJ 446479791e refactor(iam): remove legacy OIDC federation paths (#8433)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-09 07:11:50 -07:00
Chris 99af59a914 Merge commit from fork 2026-10-09 21:43:53 +08:00
Chris b77bd26874 Merge commit from fork 2026-10-09 21:43:08 +08:00
Chris 956c1cf027 Merge commit from fork 2026-10-09 21:40:59 +08:00
Chris 8b9d44110e Merge commit from fork 2026-10-09 21:40:08 +08:00
Chris 44ce8d114f Merge commit from fork 2026-10-09 21:39:18 +08:00
Chris 769b0d7578 Merge commit from fork 2026-10-09 21:28:19 +08:00
Chris 03e775947d Merge commit from fork 2026-10-09 21:18:27 +08:00
overtrue 1f7cc593e7 test(ecstore): isolate tier-free recovery fixture startup
The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.
2026-10-09 19:29:50 +08:00
Hauser 25e73d3bd6 fix(zip): statically link liblzma on macOS (#8434)
* fix(zip): statically link liblzma on macOS

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(zip): pin macOS liblzma minimum to 0.4.8

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(admin): route bucket recovery errors through facade

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): group version pagination markers

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): avoid cloning copy purge identity

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-09 16:25:24 +08:00
overtrue 1d5a50f094 fix: remove redundant pending closure in disk test
Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.
2026-10-09 14:44:39 +08:00
Hauser a464b45367 fix(rustfs): keep recovery handler within S3 gateway (#8429)
* fix(rustfs): route recovery errors through S3 gateway

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): satisfy strict Clippy pagination checks

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): avoid cloning Copy purge identity

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test: restore one-second drive request budgets

* fix(ecstore): resume pending lock bucket creation

Allow retries to reuse an uncommitted Object Lock bucket intent when the same incarnation remains unretired and all physical volumes are absent. Isolate the recovery entrypoint test from the shared production leader lock.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test: provision stack for deep catalog test futures

Apply the existing 4 MiB test-thread stack setup to table-catalog handler/store tests and the relocated-pool GET regression in both local and CI nextest profiles. These cases overflowed libtest's 2 MiB default stack on Linux.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-09 14:08:12 +08:00
overtrue 53245be9f4 fix: repair integration checks and IPC test readiness
Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.
2026-10-09 13:09:50 +08:00
rustfs-cla-assistant[bot] c47432e353 Merge remote-tracking branch 'origin/main' into gateway/integration 2026-10-09 02:20:19 +00:00
overtrue f622c1bce0 fix(test): compare legacy body wire parity as raw bytes
Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.
2026-10-09 08:42:46 +08:00
overtrue a99d8f4f89 test(connect): restore one-second drive request budgets
Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.
2026-10-09 08:21:15 +08:00
Chris 4d15b81623 refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)
T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).
2026-10-08 15:59:34 -07:00
Chris 836d30192f feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)
rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).
2026-10-08 15:05:16 -07:00
Chris 860c61ee6d feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)
* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface
2026-10-08 11:58:55 -07:00
Chris f600af4704 test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)
Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740
2026-10-08 10:47:10 -07:00
overtrue 9e747e8475 ci(footprint): count only s3_error! lines still on s3s's macro
s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.
2026-10-09 01:03:21 +08:00
Chris 96fc067765 refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)
* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.
2026-10-08 09:30:37 -07:00
overtrue 5972a09cc3 test: restore one-second drive request budgets 2026-10-08 23:55:57 +08:00
Chris e809b4899e refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)
Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734
2026-10-08 08:43:06 -07:00
Chris 584f03f300 test(ecstore): reach the bucket-config codec through the test facade (#8428) 2026-10-08 07:12:19 -07:00
Chris 8a5e730bc9 fix(storage): warn on stalled local disk operations (#8395) 2026-10-08 21:50:31 +08:00
Chris d52d6fade7 ci: make the s3s footprint baseline exact and lower-only (#8403)
* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734
2026-10-08 06:48:57 -07:00
Chris 3623d6db2d ci: raise the test-count baseline to the integration tree (#8426) 2026-10-08 06:48:40 -07:00
Hauser 1df874b724 fix(obs): pass RustFS build version to telemetry (#8425)
fix(obs): align telemetry version with build identity

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-08 21:27:24 +08:00
eye-gu 68dcad8387 fix(listing): continue versioned delimiter listings from raw key (#8424)
* fix(listing): continue versioned delimiter listings from raw key

Versioned listings with a delimiter could end a page with
IsTruncated=false while keys remained past the folded scan window.
Continue from the last raw scanned key; a folded prefix cannot
advance forward_past on versioned walks.

Fixes #8423

* fix(listing): filter resumed version prefixes

Apply the ListObjectVersions CommonPrefixes key-marker boundary before
page truncation and resume at the last raw key represented by the final
returned prefix.

Refs rustfs/backlog#2791

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: RustFS <hello@rustfs.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-08 21:26:57 +08:00
Hauser 8474feaa26 fix: reconcile bucket generations and observe lock failures (#8421)
* fix: reconcile bucket generations and observe lock failures

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix: fence bucket recovery through cancellation and quota reuse

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): route tests through storage API boundary

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): use typed lock errors in recovery

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-08 21:26:12 +08:00
Chris 484f1254ca test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)
Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734
2026-10-08 21:25:56 +08:00
Chris c9c16efbb6 ci: add the per-crate test-count ratchet (#8404)
Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734
2026-10-08 21:25:42 +08:00
cxymds d386f7741d fix(lifecycle): coordinate tier free-version recovery scans (#8387)
* fix(lifecycle): coordinate tier free-version recovery scans

* fix(lifecycle): remove obsolete recovery enqueue entry point

* test(upgrade): wait for writers after each rolling replacement
2026-10-08 21:24:34 +08:00
GatewayJ 5b4432620e refactor(iam): separate OIDC authentication mapping (#8416) 2026-10-08 21:21:22 +08:00
Chris 1a450bf228 ci(e2e): require fresh distributed JUnit reports (#8384)
* ci(e2e): require fresh distributed JUnit reports

* test(e2e): emit JUnit from successful workflow mocks
2026-10-08 06:21:19 -07:00
Chris 371241afcb test(connect): restore subscriber readiness deadlines (#8285) 2026-10-08 06:21:11 -07:00
Chris 61b31e7b45 test(connect): reject unknown health flags (#8142) 2026-10-08 06:21:03 -07:00
Chris 0d1d1487f3 test(ecstore): cover mixed metadata arrival orders (#8385) 2026-10-08 06:05:23 -07:00
Chris 3824779520 test(ecstore): cover late GET object generation changes (#8383) 2026-10-08 06:05:03 -07:00
cxymds a9670ebb4d fix(ci): preserve heal quorum retries and synchronize background tests (#8422) 2026-10-08 05:03:33 -07:00
Chris f2a59073ca feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)
`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734
2026-10-08 04:11:00 -07:00
cppudge 99ae115731 fix(ecstore): make deletes cancellation-safe and reclaim their residue (#8378)
* fix(ecstore): keep object deletes running when the caller is dropped

DeleteObject and DeleteObjects commit on every disk and then run a
separate cleanup pass, both inside the request future. When the client
disconnects, hyper drops that future and the delete stops between the
two passes: xl.meta is gone, but the rollback backup and the marked data
dir stay on disk, hidden from listings and never reclaimed (#6898).

Run the ECStore delete entry points on a detached task tracked by the
instance context. The task owns the object locks, both disk passes and
any rollback, so dropping the caller only stops the wait. Server
shutdown waits up to 30s for detached deletes after the HTTP drain.

Two smaller leaks of the same kind: a cancelled data-dir delete no
longer leaves its snapshot-lease entry marked as deleting forever, and
DiskStore::delete_versions releases its waiting slot through a guard.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* fix(ecstore): finish committed delete residue on repeated deletes

Residue that an older build stranded (#6898) has no xl.meta, so deleting
the key again reports it missing and leaves everything in place. On a
disk, a delete-data.<T> marker in one of the object's data dirs proves
that transaction T removed the metadata there and committed: the data it
marked can no longer be read, healed or rolled back on that disk.

When delete_versions or delete_version finds no xl.meta, remove what the
cleanup pass of every transaction proven that way would have removed,
under the object's metadata lease. The rollback backup goes before the
marked data dirs, so an attempt cut short still leaves the marker for
the next one. Reserved markers, rollback backups without a committed
marker and directories that also hold xl.meta or other keys are left
alone. The disk result stays FileNotFound; the extra cost is one
directory read per missing key inside the disk call that already runs,
with no new RPC, and the read stops at the first child that is not a
data dir. Finished transactions are counted in
rustfs_disk_delete_residue_finished_total instead of logged per object.

A single DeleteObject in an unversioned bucket answers a missing key
from the metadata read quorum without reaching the disks, so existing
residue is reclaimed by DeleteObjects or by the empty-listing purge.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* fix(ecstore): purge the rollback backup of a committed delete on listing

After an empty listing, the orphan purge (#7955, #8261) already removes
data dirs that carry a committed delete-data.<T> marker, but it treated
the <object>/<T>/xl.meta.bkp rollback backup of the same delete as data
and blocked the whole object directory and its ancestors. Residue left
by an abandoned delete (#6898) therefore survived the listing, and
DeleteBucket kept failing with BucketNotEmpty.

Treat a UUID dir that holds only xl.meta.bkp as purgeable when a
purgeable data dir of the same object on the same disk carries the
committed marker of that transaction. A backup without such a marker
still blocks as before, since it may be needed to roll the delete back.
The other purge gates are unchanged: every disk of the set must be
online, and the backoff applies. xl.meta or unmarked data on any disk
blocks that directory and its ancestors on every disk. Like a marked
data dir before, a backup below such a directory is still removed from
a disk that holds no xl.meta there and proves the transaction with its
own marker.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* fix(ecstore): move caller-held delete locks into the detached delete

A force (recursive) delete takes the bucket lifecycle write lock in the
S3 handler and passes only its fence down, so the store does not take
that lock again. Since the delete runs on a detached task, dropping the
request released the lock while the delete kept running: a PUT under
the same prefix could succeed and then be removed by the old delete.
Lifecycle expiry had the same shape with its table-bucket publication
read lock, of which only the lock-lost signal reaches the delete.

Add ECStore::delete_object_with_tier_delete_journal_and_guards. It
moves the caller's guards into the detached task and releases them
after the delete and its listing bookkeeping. The force-delete handler
and lifecycle expiry hand their guards over; DeleteObjects holds none.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* fix(startup): keep unclean-shutdown markers while deletes still run

Shutdown waits up to 30s for detached deletes, but it cleared the
unclean-shutdown markers even when that wait timed out. A delete then
cut short by the runtime teardown went unnoticed: the next startup
skipped the erasure-set heal that the marker triggers.

Clear the markers only when the heal handoff completed and every
detached mutation finished, and log that they are kept otherwise. Wait
after the FTP, FTPS, SFTP and WebDAV servers have stopped as well,
since their deletes take the same detached path.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

* test(ecstore): simulate tier-delete crashes below the detached delete

The chunk crash-boundary tests model a crash by aborting the task that
runs delete_object_with_tier_delete_journal while it is paused at a
TierDeleteChunkTestBarrier. That entry point now runs the delete on a
detached task, so aborting the caller no longer stops it: the delete
resumed on the old store once the barrier was dropped, and building the
restarted store on the same disks tripped the tier-delete journal proof
assertion (tier_delete_chunk_crash_boundaries_resume_without_skipping_sources
failed in CI).

Run the delete through handle_delete_object_with_journal, the body
below the detached task, so the abort stops it at the barrier as before.
tier_delete_chunk_missing_child_with_journals_fails_closed used the same
pattern; its resumed delete only failed fast because the tier rejects
removes, so it gets the same change.

Signed-off-by: Petr Slavyagin <disident47@gmail.com>

---------

Signed-off-by: Petr Slavyagin <disident47@gmail.com>
2026-10-08 02:58:14 -07:00
Chris c1af94d1d1 refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)
Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734
2026-10-08 02:57:47 -07:00
cxymds 545e794945 fix(admin): resolve event destination subscription target types (#8386) 2026-10-08 02:56:49 -07:00
cxymds 855586255f feat(auth): allow custom secret keys beyond 40 bytes (#8419)
Reuse the existing UTF-8 byte minimum for credential creation so long service account secrets remain intact. Keep generated secrets at 40 bytes and document older receiver and import constraints.
2026-10-08 17:37:15 +08:00
Hauser bf47a919ac feat(quota): add sharded durable reservation protocol (#8402)
* feat(quota): add sharded durable reservation protocol

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(quota): preserve typed task error context

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(quota): initialize all durable reservation shards

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(quota): persist bootstrap shard eligibility

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(quota): reuse allocator bootstrap state

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(connect): widen inventory startup wait

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(quota): reserve protocol v3 for unknown fixtures

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(quota): reload allocator after concurrent credit refills

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-08 16:58:32 +08:00
GatewayJ bdd23da034 refactor(iam): separate federated provider views (#8415) 2026-10-08 16:53:17 +08:00
Valentin Belyakov 89a277c3d9 feat: support object lock remaining retention days (#8407) 2026-10-08 16:52:27 +08:00
Henry Guo 3e87004032 fix(table-catalog): reuse held entry locks in warehouse scans (#8418) 2026-10-08 01:01:56 -07:00
Henry Guo 76386572f6 fix(table-catalog): drop migrated tables after rename (#8417) 2026-10-08 00:07:24 -07:00
Hauser 34917c7947 fix: resume stalled GET bodies and classify transient storage failures (#8414)
* fix: resume stalled GET bodies

Retry a stalled source through the existing identity-fenced resume budget so transient stalls do not truncate committed responses.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix: preserve data usage read quorum errors

Apply the existing system-volume error mapping to snapshot body reads so a metadata-volume outage remains a retryable quorum failure.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix: handle transient heal lock failures

Align erasure-set heal classification with the shared lock taxonomy and avoid duplicate error logging for retried scanner journal writes.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-07 23:11:36 -07:00
GatewayJ f0b2080e0c refactor(iam): add verified federated identity model (#8413) 2026-10-07 23:10:34 -07:00
Chris f84bbfe28f feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411) 2026-10-07 21:01:39 -07:00
rustfs-cla-assistant[bot] 378a2ff7f8 Merge remote-tracking branch 'origin/main' into gateway/integration 2026-10-08 02:18:57 +00:00
Hauser 9c6b0e182b fix: stabilize scanner leases and connect diagnostics (#8394)
* fix(scanner): scope observational publication leases

Separate ordinary namespace commit churn from data-movement fences and retry transient lease acquisition conflicts. Bind observational leases to the observed usage object while preserving strict authoritative publication checks.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): expect namespace admission conflict

Assert the retryable namespace-specific result when a commit finishes while strict lease admission awaits its snapshot.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(connect): add headroom to real performance probes

Give real filesystem and object integration probes enough bounded time on slow CI runners, and keep cleanup independent from measurement latency. Use a genuinely unwritable Linux fixture when tests run as root.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-07 19:04:34 -07:00
Chris ab9af5523c test: record multipart target response diagnostics (#8382)
* test(e2e): capture mixed multipart failure snapshots

* test: record multipart target response diagnostics
2026-10-07 18:15:00 -07:00
Hauser b18c5afc8d feat(targets): support Kafka SASL without TLS (#8398)
* feat(targets): support Kafka SASL without TLS

Propagate the explicit Kafka transport mode through notify and audit targets while retaining TLS as the default.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): update RustFS Kafka crates to 1.4.0

Use the published Kafka clients after the SASL plaintext transport support release.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-07 16:57:09 -07:00
rustfs-cla-assistant[bot] b88b53aa6a Merge remote-tracking branch 'origin/main' into gateway/integration 2026-10-07 21:18:17 +00:00
Hauser df988cfd86 fix(obs): export bucket gauges from live snapshots (#8399)
* fix(obs): export bucket gauges from live snapshots

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(obs): collapse snapshot update condition

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-08 02:12:48 +08:00
Henry Guo 04bc5dffd6 fix(table-catalog): preserve credential vending after rename (#8400) 2026-10-08 02:10:56 +08:00
Chris 681c95f359 ci: add daily main sync for the gateway integration branch (#8401)
Merge main into gateway/integration every weekday with a --no-ff merge
commit; abort a conflicting merge and open a sync/<source>-<date> pull
request listing the conflicting files instead. Run ci.yml on pull requests
that target gateway/integration so the branch ruleset's required checks can
report, and document both in the CI gate matrix.

Refs rustfs/backlog#2736, rustfs/backlog#2734
2026-10-08 02:08:57 +08:00
Hauser 32d98ff8b2 feat(scanner): admit background disk I/O (#8389)
* feat(scanner): admit background disk I/O

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(startup): place tests after module items

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-08 01:01:55 +08:00
cxymds 6b1554003e fix(heal): defer replacement retries until dangling grace expires (#8388) 2026-10-07 07:04:12 -07:00
GatewayJ 19978b2cb6 test(iam): freeze OIDC federation boundaries (#8393) 2026-10-07 20:05:10 +08:00
Lindroos Hsu db2abcf0a6 feat(http3): add opt-in QUIC transport (#7903)
* build(http3): add optional s3s-http3 dependency

Signed-off-by: tommady <tommady@users.noreply.github.com>

* feat(server): add opt-in HTTP/3 transport

Signed-off-by: tommady <tommady@users.noreply.github.com>

* test(server): cover HTTP/3 TLS configuration

* feat(server): add opt-in HTTP/3 support

Serve S3 requests over QUIC with the existing middleware stack.

Signed-off-by: tommady <tommady@users.noreply.github.com>

* fix(http3): cancel pending body reads without panicking

* test(http3): keep transport independent of s3s

---------

Signed-off-by: tommady <tommady@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-07 03:59:21 -07:00
Hauser 14eb28dcd6 chore(deps): refresh workspace dependencies and hotpath profiling (#8380)
* chore(deps): update workspace dependencies

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix: allow hotpath label marker attributes

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix: refine hotpath profiling attribution

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test: guard object hotpath attribution

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* Add ProfileReasonCode to CPU profile capture

Signed-off-by: Hauser <housemecn@gmail.com>

* fix: restore CPU profile test import

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-07 03:02:39 -07:00
ormandj 2f0143ef15 fix(ecstore): skip warning for absent .minio.sys volume (#8373)
* fix(ecstore): skip warning for absent .minio.sys volume

Startup MinIO migrations read from .minio.sys and treat its absence as expected, but resolve_read_all_result logged a disk_local_read_version_fallback warning for every such read. Keep the access check and the VolumeNotFound return, and skip only the warning for the migrating metadata volume.

Refs #8372

* test(ecstore): cover .minio.sys descendants and lookalike volumes

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-07 01:02:47 -07:00
Henry Guo 3b92708011 feat(table-catalog): add durable catalog backups (#8266)
* feat(table-catalog): add durable catalog backups

* fix(table-catalog): preserve migration lock ordering

* fix(table-catalog): preserve backup recovery fences

* fix(table-catalog): order backup publication locks

* fix(table-catalog): preserve recovery publication fences

* test(table-catalog): use the storage facade for recovery fencing

* test(ecstore): repair catalog publication fixtures

* fix(table-catalog): address catalog backup clippy failures

* test(table-catalog): adapt generation races to publication fencing

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-06 23:53:53 -07:00
Chris 8fea8ef600 test(e2e): bound mixed-version multipart acknowledgement (#8381) 2026-10-07 14:08:47 +08:00
GatewayJ 477f768f7e feat(table-catalog): support disabling empty catalogs (#8376)
* feat(table-catalog): support disabling empty catalogs

* fix(admin): use the storage error domain module

* style(admin): format the publication fence error

* fix(table-catalog): use shared API error constructors
2026-10-07 14:07:56 +08:00
shurkus 47ed6717de fix(tier): drop reserved internal headers from transition PUT (#8363)
* fix(tier): drop reserved internal headers from transition PUT

* test(tier): transition PUT against a target rejecting reserved headers

The fake S3 target gains a mode that answers 400 InvalidArgument to any
PutObject or UploadPart carrying an x-minio-internal-* request header,
the rule MinIO and the Storj gateway enforce. RT-14 transitions an
inline and a non-inline object through a real server to that target and
asserts the stored bytes, both transition identity pairs and read-back.

The e2e-full linux selection digest is refreshed for the new test.

---------

Co-authored-by: Vector <vector@wcomp.io>
Co-authored-by: cxymds <cxymds@gmail.com>
2026-10-07 14:06:14 +08:00
Chris 480312e5ad fix(ecstore): refresh parsed bucket configs on every load (#8379) 2026-10-06 22:59:14 -07:00
yihong 1c5bbc0c47 ci: do not run some actions on fork (#8318)
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
2026-10-06 22:02:55 -07:00
Chris 14adc5ed3f test(ecstore): cover real metadata cache and late identity reads (#8371)
* test(ecstore): cover real metadata cache and late identity reads

* test(e2e): bound max-keys fixture upload concurrency
2026-10-06 22:02:48 -07:00
dependabot[bot] 714706328b build(deps): bump rustls from 0.23.42 to 0.23.45 in /fuzz in the cargo group across 1 directory (#8370)
build(deps): bump rustls in /fuzz in the cargo group across 1 directory

Bumps the cargo group with 1 update in the /fuzz directory: [rustls](https://github.com/rustls/rustls).


Updates `rustls` from 0.23.42 to 0.23.45
- [Release notes](https://github.com/rustls/rustls/releases)
- [Changelog](https://github.com/rustls/rustls/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rustls/rustls/compare/v/0.23.42...v/0.23.45)

---
updated-dependencies:
- dependency-name: rustls
  dependency-version: 0.23.45
  dependency-type: indirect
  dependency-group: cargo
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-06 20:05:36 -07:00
Chris cd0b895ad7 fix(protocols): share the server context with protocol storage (#8367)
* fix(protocols): bind storage authorization to the startup server context

* test(protocols): publish the context through the runtime boundary
2026-10-06 19:58:58 -07:00
Chris 073b4d838e test(ecstore): verify multi-generation authority model (#8339) 2026-10-06 19:58:13 -07:00
Hauser 95268a3b9e fix(obs): avoid reactor panic in OTLP periodic exports (#8368)
fix(obs): use blocking clients for OTLP exporters

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-06 23:31:56 +08:00
Chris f7c4ac11b2 fix(heal): retry typed RPC transport failures after peer restart (#8365)
* fix(heal): retry typed RPC transport failures after peer restart

* test(heal): keep original errors in GOAWAY clone coverage
2026-10-06 08:11:19 -07:00
Sven Gottwald 9d9e92bab8 fix(compose): Refactor volume paths and add service_completed_success… (#8355)
fix(compose): Refactor volume paths and add service_completed_successfully

The volume paths for the volume-permission-helper service have been updated and the command has been adjusted to reflect the new paths.

Ensure that the volume-permission-helper service runs successfully before starting the rustfs service.

Signed-off-by: Sven Gottwald <2502366+svengo@users.noreply.github.com>
2026-10-06 22:42:15 +08:00
overtrue 0ad0a6d8de fix(heal): increase stack size for absence receipt replay test
The  test used
 with default stack size, but
the 16-disk deep heal path overflows the default macOS thread stack
(~8MB). Switch to an explicit  with 32MB stack
and manual tokio runtime, matching the pattern used by sibling tests
in the same file.

Verified: test passes in isolation after the change.
2026-10-05 04:10:55 -07:00
GatewayJ 051a54f129 fix(table-catalog): verify Iceberg metadata path signatures (#8291)
* fix(table-catalog): verify Iceberg metadata path signatures

* fix(deps): allow pinned s3s Git source
2026-10-05 02:03:41 -07:00
dependabot[bot] c71e4f72ea build(deps): bump the dependencies group with 4 updates (#8359)
Bumps the dependencies group with 4 updates: [aws-sdk-kms](https://github.com/awslabs/aws-sdk-rust), [aws-sdk-s3](https://github.com/awslabs/aws-sdk-rust), [aws-sdk-sts](https://github.com/awslabs/aws-sdk-rust) and [hotpath](https://github.com/pawurb/hotpath-rs).


Updates `aws-sdk-kms` from 1.122.0 to 1.123.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `aws-sdk-s3` from 1.151.0 to 1.152.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `aws-sdk-sts` from 1.118.0 to 1.119.0
- [Release notes](https://github.com/awslabs/aws-sdk-rust/releases)
- [Commits](https://github.com/awslabs/aws-sdk-rust/commits)

Updates `hotpath` from 0.28.0 to 0.28.2
- [Release notes](https://github.com/pawurb/hotpath-rs/releases)
- [Changelog](https://github.com/pawurb/hotpath-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pawurb/hotpath-rs/compare/v0.28.0...v0.28.2)

---
updated-dependencies:
- dependency-name: aws-sdk-kms
  dependency-version: 1.123.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: aws-sdk-s3
  dependency-version: 1.152.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: aws-sdk-sts
  dependency-version: 1.119.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: hotpath
  dependency-version: 0.28.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-05 10:38:53 +08:00
Hauser 003570b61e fix(heal): bound durable MRF rescheduling (#8351)
Reuse stable live-owner proof anchors to avoid duplicate in-flight admission. Pace unverified completions and unavailable attempts with bounded jitter while retaining checkpoints, exact proofs, and generation isolation.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-04 09:04:11 -07:00
Chris 0c3efda7bd docs: refresh security advisory lessons (#8350) 2026-10-04 22:48:56 +08:00
Hauser 2c7e7d195c fix(auth): return ExpiredToken for retained STS credentials (#8344)
* fix(auth): return ExpiredToken for retained STS credentials

Validate session identity independently of credential expiry in both IAM
rejection and claims validation, and preserve typed S3 errors at the caller.
Keep token binding, disabled-account precedence, and service-account behavior.
Add exact expiry, IAM, and HTTP/XML regression coverage.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): synchronize the metadata slowtail hedge regression

Park the slow metadata task at a test-only fault checkpoint and release
healthy reads after the fault is active. Assert the spare response and
cancelled-task drain without imposing a 300 ms disk/scheduler deadline.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ilm): synchronize copy compensation transition stages

Observe automatic compensation reaching the mock tier PUT checkpoint before
waiting for the final metadata commit. Assert forced enqueue failure,
automatic backfill, one completed tier upload, and payload integrity.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-04 06:16:40 -07:00
Hauser 0764785754 perf(scanner): reduce scan and checkpoint peak allocations (#8346)
* perf(scanner): reduce scan and checkpoint peak allocations

* test(scanner): isolate partial commit recovery scenarios
2026-10-04 06:04:19 -07:00
Chris 182b2bfe98 fix(s3): encode multipart listing response keys (#8347) 2026-10-04 20:24:26 +08:00
cxymds 00a549504c test: allow cleanup fixture CI headroom (#8336) 2026-10-04 20:20:05 +08:00
Hauser 83fa2611f4 fix(ecstore): own listing workers through cancellation (#8342)
Consolidate the three page collectors, preserve completed worker failures, and abort nested merges when their owner is dropped.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-04 02:04:01 -07:00
hector e22414fe68 ci: coerce the pool timeout input to a number
timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes
is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to
the string '240' and the runner rejected the template: 'An error occurred
when attempting to determine the step timeout ... Unexpected value 240'
(chain run 36505625402 job at 02:25:52, standalone run 36393453114 at
07:46:37). Both pool runs since #8155 merged have therefore executed
without the intended step budget. Wrap the expression in fromJSON and
update the policy test that pinned the broken literal.
2026-10-04 01:45:12 -07:00
hector 45f97bccf9 ci: coerce the pool timeout input to a number
timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes
is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to
the string '240' and the runner rejected the template: 'An error occurred
when attempting to determine the step timeout ... Unexpected value 240'
(chain run 36505625402 job at 02:25:52, standalone run 36393453114 at
07:46:37). Both pool runs since #8155 merged have therefore executed
without the intended step budget. Wrap the expression in fromJSON and
update the policy test that pinned the broken literal.
2026-10-04 01:45:12 -07:00
Chris 046fb21d86 fix(test): restore authenticated cluster fault scenarios (#8340)
* fix(test): restore authenticated cluster fault scenarios

* fix(test): use the E2E storage API boundary
2026-10-04 01:03:59 -07:00
Chris 4783adff36 test(ecstore): enforce main core regression gates (#8338)
* test(ecstore): enforce main core regression gates

* test(ecstore): pin external shards and completion regressions
2026-10-04 01:03:42 -07:00
GatewayJ 68ad1a1d7b fix(protocols): enforce S3 policies for file protocol operations (#8262) 2026-10-03 21:40:56 -07:00
overtrue 7c289001ed test(connect): isolate short HTTP fixtures in nextest 2026-10-03 21:40:18 -07:00
overtrue 7b1ffbfdc1 fix(ecstore): hide rollback backup residue in prefix listings 2026-10-03 21:40:18 -07:00
Chris 562e03670e fix(storage): retain typed owners across borrowed writes (#8329)
* fix(storage): retain typed owners across borrowed writes

* fix(storage): preserve Copy write completion policy

* refactor(storage): remove redundant write owner references
2026-10-03 20:02:39 -07:00
Hauser 5e1bd498ce fix(scanner): isolate and coalesce dirty usage journals (#8325)
* fix(scanner): isolate and coalesce dirty usage journals

Bind dirty replay records to a stable node owner so peers cannot overwrite
or delete another node's pending journal. Replace the unbounded command
queue with bounded coalesced state and revision-aware background saves.
Keep legacy records isolated and enforce replay byte limits with conservative
whole-bucket or unverified fallback. Add publication-stage diagnostics and
storage regressions for ownership, clearing, retry, and restart behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): collapse confirmed journal clear condition

Use a let-chain to retain the generation fence before removing a confirmed
bucket and updating its scope byte accounting. Satisfy collapsible_if
without changing journal clearing behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-04 03:02:22 +08:00
Chris 5aca750a6e fix(listing): count logical prefixes before pagination (#8328)
* fix(listing): count logical prefixes before pagination

* test(listing): normalize the fixed corpus seed
2026-10-04 01:24:39 +08:00
Chris a760d567b6 test(storage): make commit fault checkpoints independent (#8326)
* test(storage): make commit fault checkpoints independent

* test(storage): simplify commit checkpoint lookup
2026-10-04 01:19:28 +08:00
Chris 47d208325e test(storage): verify rejected rollback inspection on real disks (#8331) 2026-10-04 01:13:39 +08:00
Chris d087416ff9 refactor(storage): separate metadata scheduling from quorum decisions (#8327) 2026-10-04 01:04:37 +08:00
Chris 309128f034 refactor(storage): bind rename rollback to its commit context (#8324) 2026-10-03 23:55:33 +08:00
Nils Melchert 7a0f8561b6 fix(oidc): ignore groups without a matching policy at login (#8164)
* fix(oidc): ignore groups without a matching policy at login

OIDC login failed with "OIDC policy mapping did not resolve to current
policies" whenever any mapped group lacked a policy of the same name.
Directory-backed providers such as Active Directory always emit groups
like `DOMAIN\Domain Users` that can never be mapped, so group-based
authorization was impossible there.

Keep only policy names that resolve to an existing policy and are valid
in session claims, and reject the login only when none remain. The
signed `policy` claim still lists only resolved names, so request-time
evaluation and site replication receivers keep their invariant.

Refs #8163

* fix(oidc): only ignore unmapped groups-claim values

Limit the relaxed resolution to policy names derived solely from the
groups claim. Names from ROLE_POLICY, a dedicated CLAIM_NAME claim, or
an explicit IAM policy mapping must still all resolve.

Reject the login when a mapped name refers to an existing policy whose
name is not allowed in session claims: dropping it could remove an
explicit Deny and broaden access.

Claim-unsafe names are only checked against the in-memory policy cache
and never passed to storage-backed policy loading.

Refs #8163

* docs(oidc): document ignorable roles claim values and cover wiring

Roles claim values are merged into the canonical groups, so values
without a matching policy are ignored like groups-claim values. This
covers built-in provider roles such as Keycloak's `offline_access`.
Document that in the field and method docs and in the provider
requirements, and pin it with a test.

Assert that the OIDC authorization carries the group claim policies so
a regression in the wiring cannot silently disable the relaxation.

Refs #8163

* fix(oidc): ignore group policies only after confirmed absence

merge_policies drops names whose load fails, so a storage or decode
error for an uncached group policy was indistinguishable from a missing
one. The group name was then ignored and the session signed without it,
which could remove an existing Deny that request-time checks cannot
restore.

Add IamSys::policy_exists, which consults the cache and then storage and
reports false only for NoSuchPolicy while returning every other error.
The OIDC binding now ignores a group-derived name only after a confirmed
absence and rejects the login when a lookup fails.

Refs #8163

---------

Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-03 22:12:51 +08:00
Chris e77c76272a docs(storage): verify object generation authority contract (#8322) 2026-10-03 21:30:54 +08:00
Chris c0e63ee543 docs: trim redundant project skills (#8323) 2026-10-03 21:30:26 +08:00
Chris 8452910895 fix: resolve pagination fixtures and heal recovery CI failures (#8316)
* test(e2e): bound delimiter pagination fixture uploads

* fix(heal): classify CAS lock failures before publication

* fix(heal): retry unproven absence within the object budget

* docs(heal): clarify object retry classification

* test(s3): seed the delimiter fixture with bounded concurrent PUTs
2026-10-03 20:40:52 +08:00
Chris 5efe4c5703 test(connect): verify local runtime peer credential boundary (#8304) 2026-10-03 17:19:02 +08:00
AL afa367d220 fix(http): classify peer closures from error sources (#8307)
fix(http): classify peer transport closure by error source

Co-authored-by: Codex <codex@bednarowski.ca>
2026-10-03 17:18:41 +08:00
AL c6914f4132 fix(scanner): drain terminal remote responses to clean EOF (#8306)
fix(scanner): drain terminal remote responses to EOF

Co-authored-by: Codex <codex@bednarowski.ca>
2026-10-03 17:18:18 +08:00
RJ Regenold 48a6104aad feat(iam): expose stored bucket tags to OPA (#8298)
* feat(iam): expose stored bucket tags to OPA

* fix(ci): lint bucket tag fixtures and register new e2e tests

* fix(ecstore): restore strict Clippy compatibility on Rust 1.99

Use try_update without changing atomic ordering or overflow behavior. Keep
recursive storage futures boxed once at each frame and remove the redundant
async-recursion macro, including its non-recursive SQL planner use. Remove
needless closure borrows and orphaned dependency entries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 537c15277a)

* fix(deps): replace yanked yoke-derive release

(cherry picked from commit b54b32b8a9)

* perf(iam): borrow unchanged OPA bucket tag conditions

Reuse clean conditions for bucketless and untagged OPA decisions. Keep the existing clone-and-sanitize path for tagged requests, reserve enrichment capacity, and preserve authoritative lookup errors. Cover borrowed input, reserved namespaces, immutable request state, and both paths in the OPA input regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(iam): clarify OPA bucket policy fallback semantics

Document that an OPA tag mismatch remains an implicit IAM denial which a bucket-policy Allow can supplement. Extend the existing S3 contract test to contrast a normal OPA denial with corrupt bucket metadata, whose lookup error must abort the same Allow fallback.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): isolate durable progress fixtures from competing IO

Reserve the nextest execution budget for schedule, streaming lock, retirement, MRF checkpoint/crash, and native scanner progress scenarios. Preserve their internal concurrency, existing deadlines, typed failure assertions, and zero-retry policy in default and CI profiles.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: rjregenold <214054+rjregenold@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-03 17:18:04 +08:00
chapman 4804162be8 fix(s3): correct ListParts pagination termination and markers (#8295)
* fix: emit NextPartNumberMarker only when ListParts is truncated

ListPartsInfo.next_part_number_marker was a non-optional usize that
defaulted to 0 and was only assigned when the response was truncated.
The S3 serializer then emitted it unconditionally as Some(0), causing
AWS SDK paginators to loop infinitely on part_number_marker=0 instead
of terminating.

Change the field to Option<usize> (None by default) and set it only
inside the is_truncated branch. The S3 output layer now uses
.and_then() so NextPartNumberMarker is absent when IsTruncated=false,
matching AWS S3 behavior.

Fixes #8208

* fix(s3): honor sparse ListParts markers and verify termination

Resume part listings at the first part above the numeric marker, even
when that marker is absent. Use binary search over the sorted part
numbers and retain the existing exact-tail empty-slice path.

Add storage, XML, and real AWS SDK paginator regressions for empty and
terminal pages, sparse markers, and multipart completion integrity.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): register ListParts E2E in selection manifests

Register the intentional paginator regression in the smoke selection
and Linux full-suite inventory. Audit the actual CI test identities
and verify that removing the new case restores both prior digests.

Keep the profile filters, strict digest checks, and Darwin full entry
unchanged. The latter has an independent pre-existing selection drift.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-03 17:12:26 +08:00
Jason Kossis 6c376413bb fix(scanner): preserve checkpoint children in large prefixes (#8292)
* fix(scanner): preserve checkpoint children in large prefixes

* fix(ecstore): restore strict Clippy compatibility on Rust 1.99

Use try_update without changing atomic ordering or overflow behavior. Keep
recursive storage futures boxed once at each frame and remove the redundant
async-recursion macro, including its non-recursive SQL planner use. Remove
needless closure borrows and orphaned dependency entries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 537c15277a)

* fix(deps): replace yanked yoke-derive release

(cherry picked from commit b54b32b8a9)

* chore: keep scanner repair separate from CI corrections

* test(scanner): verify large-prefix resume and compaction lifecycle

Exercise Normal and Deep scans across cycle and leader changes. Validate
persisted frontiers, bound replay to interrupted boundary objects, and
prove that a completed same-plan sweep restores compaction without losing
object or byte totals.

Refs: rustfs/backlog#2710

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-10-03 17:01:16 +08:00
Hatsune Imagine f5956999a7 feat(webdav): report quota properties on PROPFIND (#8197)
dav-server queries get_quota once per PROPFIND without a path, so the
driver now reports a single session-scoped (used, total) pair:

- when every bucket visible to the session has a hard quota: the summed
  cached bucket usage against the summed quota limits (a bucket whose
  usage cache has no scanner snapshot yet counts as zero);
- otherwise: the cluster's usable capacity, the same erasure-aware
  used/total the console dashboard shows, cached for 30s since WebDAV
  clients poll quota and storage-info fans out to every disk;
- when neither source is available the properties are omitted from the
  response, which is the previous behavior.

All inputs are cache or snapshot reads; reporting never triggers a
scanner cycle or a live listing. Backend failures surface as
FsError::GeneralFailure instead of a bogus zero capacity.

Adds a session_capacity_view hook to the protocols StorageBackend trait
(default: no capacity support, keeping quota properties omitted), its
rustfs-side implementation, aggregation unit tests, and an e2e test
covering both reporting modes.
2026-10-03 16:50:24 +08:00
Hiroaki KAWAI 94105a1044 fix(heal): skip removed usage observations during read repair (#8135)
* fix(scanner): skip absent usage observations during heal admission

* fix: skip usage observation metadata recreation

* style: apply rustfmt to scanner fix

* Revert "style: apply rustfmt to scanner fix"

This reverts commit c871d03911.

* Revert "fix: skip usage observation metadata recreation"

This reverts commit d500661be6.

* Revert "fix(scanner): skip absent usage observations during heal admission"

This reverts commit c7e03eb926.

* fix: skip read repair for usage observations

* fix(heal): skip removed usage observations during read repair

* fix(connect): return profile capture result directly

* fix(deps): replace yanked yoke-derive release

---------

Co-authored-by: cxymds <cxymds@gmail.com>
2026-10-03 16:49:19 +08:00
Chris 8df8360bc2 docs(release): maintain milestones after publishing (#8317) 2026-10-03 16:48:13 +08:00
GatewayJ 753e8aa657 perf(ecstore): reduce multipart and object I/O overhead (#8095)
* perf(ecstore): share multipart cleanup paths across disks

* perf(ecstore): avoid encode copies and cached descriptor duplication

* test(ecstore): cover default ingest selection and overrides

* test(ecstore): cover cached positioned reads in both modes

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: Chris <anzhengchao@gmail.com>
2026-10-03 14:53:00 +08:00
Chris 641c4b3493 chore(release): merge 1.0.1 back into main and refresh installation references (#8314)
* fix(ci): include pagination regression in full E2E selection

* fix(deps): replace yanked yoke-derive release

* fix(scanner): expose pause backlog replica diagnostics (#8258)

* fix(scanner): expose pause backlog replica diagnostics

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(connect): stabilize runtime profile lease cancellation

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(connect): tolerate delayed schedule startup in CI

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): retry quota reads during usage warmup

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): avoid meta-bucket incarnation self-deadlock

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(test): use persisted incarnation in heal fixture

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(usage): reconcile stale counters after lifecycle expiration (#8108)

* fix(usage): reconcile stale counters after lifecycle expiration

* fix(usage): account lifecycle expiry during continuous writes

* test(usage): run lifecycle usage scenarios on one scanner store

* test(usage): use a Windows-representable pre-mutation offset

* fix(usage): harden expiry accounting recovery and quota checks

Borrow expiry receipt bucket names and avoid allocating a map key on cache hits. Cover cancelled receipts, durable snapshot recovery, and legacy quota admission after scanner confirmation. Use representable timestamp offsets in the quota regression.

Refs rustfs/backlog#2689

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(usage): recover stale persisted counts through the scanner

Seed incorrect complete usage for empty and retained-object buckets, then run the real scanner and publication consumer without further object mutations. Verify durable and admin usage over two cycles instead of writing a corrected snapshot in the test.

Refs rustfs/backlog#2689
Refs rustfs/backlog#2691

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(e2e): bound delimiter pagination fixture concurrency

The 120-second smoke timeout expired after 1018 of 1200 serial fixture PUTs, before LIST ran. Prepare the same objects with at most eight concurrent requests and await every PUT. Retain the timeout and strengthen exact prefix, KeyCount, empty Contents, and continuation-token assertions with phase diagnostics.

Refs rustfs/backlog#2689

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(upgrade): establish a persisted previous-release baseline

Seed the pinned previous-release cluster and restart it once with its data intact before replacing any node. Require every old writer to pass the strict readiness probe and preserve the seed through both mixed phases and the final current cluster. Keep InternalError fail-fast behavior and all existing compatibility deadlines and assertions.

Refs rustfs/backlog#2689
Refs rustfs/backlog#2384

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): bound cancellation metadata persistence waits

Use the system-bucket incarnation boundary now supplied by main PR #8268. Bound the three cancellation waits that previously hung during pool.bin persistence, retaining their remote-generation, target-cohort, and durable-state assertions.

Refs rustfs/backlog#2697
Refs rustfs/backlog#2689

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(e2e): retain startup and shutdown failure diagnostics

* fix(test): supply CPU workload for sampler regression

* fix(ci): locate security chain scripts in the workspace

* fix(usage): recover historical counters with generation fencing (#8273)

* fix(usage): recover historical counters during continued writes

Use newer converged scanner snapshots to reconcile stale absolute usage
baselines while preserving concurrent mutation and expiry receipt fences.
Cover durable publication, admin and quota reads, and legacy generations.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(usage): fence snapshots and move preserved cache entries

Apply the cached scanner generation floor before every reconciliation path
and retain it even when an older snapshot happens to match core counts.
Move preserved usage entries instead of cloning their histogram maps under
the cache lock, retaining expiry receipt identity and cancellation fences.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* test(scanner): verify checkpoint takeover and repair dispatch (#8275)

* test(scanner): cover checkpoint handoff and repair dispatch

Drive runtime budget expiry, partial-cycle persistence, leadership claims,
and stale checkpoint rejection between real disk-backed fixture scans.
Verify that a metadata repair beyond the first bounded prefix is saved in
the scanner ledger and dispatched by the MRF consumer.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* ci: isolate scanner fixtures and refresh full e2e membership

Reserve nextest capacity for the real-disk scanner publication and MRF
admission fixtures. Bind both platform membership checks to the reviewed
pagination deadline test added on main.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(scanner): consolidate checkpoint fixture lifecycle

Keep one durable control store across timeout and leadership transitions,
and inject generation advancement into the shared checkpoint scenario.
Check the actual saved metadata path so late-write rejection also proves
that existing checkpoint bytes remain intact.

Centralize MRF fixture isolation and reuse nextest process isolation when
the startup environment already satisfies the test contract.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(obs): distinguish allocator counters from live memory (#8274)

* fix(obs): distinguish allocator counters from live memory

Preserve count/counter semantics and mark requested-byte attribution unavailable when live statistics or sampling are missing. Document sustained multipart memory diagnosis.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(obs): parse allocator statistics from one node

Resolve each statistic before interpreting its shape, avoiding unsupported-field tree scans and mixing data across wrapper scopes. Preserve unavailable-statistics policy and add precedence regressions.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): isolate late parity recovery from metadata hedges

Use the existing object-scoped hedge timer barrier in exact-count recovery fixtures. Preserve payload and total-read assertions and verify that the omitted parity disk is read only during late refresh.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(usage): combine identical snapshot retention branches

* fix(test): await HTTP sender readiness in Top RPC fixture

* [release/1.0.1] Gate multipart copy through write admission (#8284)

Gate multipart copy through write admission

Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* Gate multipart copy through write admission (#8283)

Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission.

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix: add UploadPart OOM validation guardrails (#8287)

* docs(release): validate candidates on release branch

* fix(scanner): validate checkpoints against global cycle fence (#8278)

## Related Issues

Related to rustfs/backlog#2701.

## Summary of Changes

Route scanner checkpoint cycle and leader validation through the global store while retaining the owning set for cache persistence, CAS revisions and publication admission.

## Verification

Two independent final-diff source reviews found no issues across correctness, concurrency and durability, test coverage, compatibility, performance and simplicity on head `8b8fe51d092090b053f552ae283960e2e306be33`. Root approval `5373624714` is bound to that exact head. Regression tests cover real two-pool routing, stale fences, post-save rejection and CAS conflicts; their reported local execution belongs to the PR author, not this merge operation. Current required CI remains pending, and this authorized admin squash does not establish CI or runtime acceptance.

## Impact

Restores checkpoint progress when global cycle and leader state differ from a set-scoped view. No format, retry, timeout, assertion or scanner-policy changes are introduced by this diff. The three prior main scanner failures remain unproved repaired.

## Additional Notes

Full validation must run on the resulting exact main revision. Reverting this patch restores the earlier set-scoped fence lookup and its checkpoint rejection behavior.

* fix(ci): restore E2E membership and pagination timeouts (#8281)

* ci: locate the auto-testing checkout for lanes that run evidence from a subdirectory (#8279)

## Related Issues

Follow-up to #8229.

## Summary of Changes

Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence.

## Verification

The exact PR head b66129ab9f passed one mechanical correctness and simplicity review, nine real-Git layout and provenance checks, and sixteen existing evidence/envelope tests. The baseline sibling layout failed with git exit 128; the corrected layout succeeded while revision mismatches and missing checkouts stayed rejected. Current PR checks are completed with successful or skipped conclusions, including the aggregate.

## Impact

Both lane and private-script revision checks remain intact. No time limits, assertions, production behavior, or evidence validation requirements change. The synthetic layout checks do not execute the actual scheduled security suite; integrated main CI and release acceptance remain separate gates.

## Additional Notes

Approved review 5374559393 is bound to the exact head above. Reverting the single-file change restores the previous checkout lookup.

* fix: add UploadPart OOM validation guardrails

Add a Docker validation harness for backlog#2704 so the ordinary UploadPart
low-concurrency memory workload can be reproduced with comparable case metadata,
process/cgroup sampling, TLS and metrics toggles, cache-env controls, and write
reclaim/direct-write experiments.

Warn when operators set the unrecognized RUSTFS_OBJECT_CACHE_* variables that
appeared in the reporter compose file. The variables are reported but remain
ignored, so startup does not silently change object data cache behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: AL <allan.bednarowski@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): pass fence store to checkpoint fixture

* fix(s3): queue bucket operations and restore strict Clippy checks (#8290)

* fix(s3): queue concurrent bucket creation and deletion

Keep eight active bucket transactions and bound admission waiting to 128 requests and 30 seconds. Preserve detached transaction ownership and return Retry-After with overload responses.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): restore strict Clippy compatibility on Rust 1.99

Use try_update without changing atomic ordering or overflow behavior. Keep
recursive storage futures boxed once at each frame and remove the redundant
async-recursion macro, including its non-recursive SQL planner use. Remove
needless closure borrows and orphaned dependency entries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* chore: refresh dependencies and atomic update APIs

Update workspace dependencies and the lockfile. Replace deprecated
atomic fetch_update aliases with try_update while preserving closures
and memory ordering.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* improve

* fix(scanner): diagnose and verify pause backlog recovery (#8293)

* fix(scanner): diagnose and verify pause backlog recovery

Expose the retained replica snapshot and claimed membership in abnormal
admin status responses. Keep diagnostics off metrics updates and verify
single-pool recovery and conflicting-proof preservation across 24 sets.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(scanner): move replica snapshots into diagnostics

Consume the terminal admin read snapshot in a single state match and move
membership, revision, and error buffers into the response. Verify buffer
handoff and the unchanged JSON contract without altering ledger authority.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* fix(heal): wait for held legacy responsibility in replay test

* ci: remove Docker Hub description sync

* fix: emit NextPartNumberMarker only when ListParts is truncated

ListPartsInfo.next_part_number_marker was a non-optional usize that
defaulted to 0 and was only assigned when the response was truncated.
The S3 serializer then emitted it unconditionally as Some(0), causing
AWS SDK paginators to loop infinitely on part_number_marker=0 instead
of terminating.

Change the field to Option<usize> (None by default) and set it only
inside the is_truncated branch. The S3 output layer now uses
.and_then() so NextPartNumberMarker is absent when IsTruncated=false,
matching AWS S3 behavior.

Fixes #8208

(cherry picked from commit 44de803a38)

* fix(s3): honor sparse ListParts markers and verify termination

Resume part listings at the first part above the numeric marker, even
when that marker is absent. Use binary search over the sorted part
numbers and retain the existing exact-tail empty-slice path.

Add storage, XML, and real AWS SDK paginator regressions for empty and
terminal pages, sparse markers, and multipart completion integrity.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 673031eea1)

* fix(storage): publish delete rollback backups atomically

Stage rollback metadata outside the rollback directory and publish it only after the full write succeeds. A short write must not leave a backup that quorum rollback can rename over acknowledged version history.

Add an isolated real short-write regression and register the backported ListParts SDK test in the smoke and Linux full inventories.

* test(e2e): register paginator regression in Darwin inventory

* fix(release): install yq before Helm template checks

* chore(release): align installation references for 1.0.1

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Peder Bergan <pederbe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: AL <allan.bednarowski@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Chapman <touch65536@gmail.com>
2026-10-03 14:40:52 +08:00
GatewayJ f07142910c feat(s3select): support quoted record delimiters (#8203)
* feat(s3select): support quoted record delimiters

* fix(s3select): preserve CSV comment limits and CR fields

* fix(s3select): preserve CRLF fields and compressed record limits
2026-10-03 11:10:11 +08:00
Henry Guo 5efa6254eb feat(table-catalog): expose durable strong diagnostics (#8232)
* feat(table-catalog): expose durable strong diagnostics

* fix(table-catalog): stabilize snapshot diagnostics and metadata scans

Reuse one recovery observation for exports and diagnostics in both backing
modes. Follow the persisted metadata directory for renamed and registered
tables, and bound diagnostics under continuous snapshot changes.

Remove redundant recovery work and fix the production dead-code and clone
lint failures without relaxing checks. Add deterministic race, retry, and
metadata reachability coverage and clarify read-only candidate semantics.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(e2e): bound pagination fixture upload concurrency

Keep the 1200-object delimiter boundary fixture while preparing it with
at most 16 concurrent PUTs. Serial preparation exhausted the existing
120-second smoke deadline before the listing request was issued.

Report fixture preparation time and verify the exact common prefixes,
empty contents, KeyCount, and absent continuation token without relaxing
the timeout, durability settings, or dataset size.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: GatewayJ <835269233@qq.com>
2026-10-03 10:00:23 +08:00
唐小鸭 5c99417073 fix(sse): guard copy-source SSE-C keys over TLS and reject bucket-keyed KMS context (#8296)
* fix(sse): count copy-source SSE-C headers in the TLS transport guard

The SSE-C transport guard only looked at the object's own customer-key
headers. A CopyObject or UploadPartCopy that read an SSE-C source into a
non-SSE-C destination carried the source key only in the
x-amz-copy-source-server-side-encryption-customer-* headers, so it was
accepted on a plaintext transport with RUSTFS_SSE_C_REQUIRE_TLS=true and
was missing from rustfs_ssec_plaintext_requests_total.

Treat the copy-source triple as SSE-C headers too.

* fix(sse): reject a KMS context key that replaces the location entry

Managed SSE wraps each data key under an encryption context that carries
{bucket: bucket/key}, added with or_insert, while only the client context
is persisted and the entry is rebuilt on read. A client context entry
keyed by the bucket name therefore replaced the location entry on write
and on every read, so the data key was no longer tied to the object's
location.

Reject such a context with 400 InvalidArgument at the single managed-SSE
write entry, before the KMS is called. The read side is unchanged, so
objects stored with such an entry stay readable, and other keys,
including other bucket names, are still accepted.

---------

Co-authored-by: Hauser <housemecn@gmail.com>
2026-10-03 09:31:08 +08:00
Chris 4aa3cde1d9 fix(connect): send endpoint origin for relay session authentication (#8310) 2026-10-03 01:06:25 +08:00
Chris 87c1c851c2 ci: remove Docker Hub description sync (#8303) 2026-10-02 20:04:45 +08:00
Hauser 370b517b4f fix: resolve Rust 1.99 Clippy warnings (#8301)
Replace deprecated atomic fetch_update calls with try_update, remove
redundant closure borrows, and drop unnecessary async-recursion macros.
Keep atomic orderings and explicitly boxed recursive calls unchanged.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-02 10:11:22 +08:00
Chris 791af1888f fix(connect): preserve site replication S3 key path (#8300) 2026-10-02 05:43:24 +08:00
Chris 50e6c907f9 fix(s3): bound multipart copy admission and HTTP write buffers (#8288) 2026-10-01 21:51:27 +08:00
hector 1efa86b8a3 ci: locate the auto-testing checkout for lanes that run evidence from a subdirectory (#8279)
## Related Issues

Follow-up to #8229.

## Summary of Changes

Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence.

## Verification

The exact PR head b66129ab9f passed one mechanical correctness and simplicity review, nine real-Git layout and provenance checks, and sixteen existing evidence/envelope tests. The baseline sibling layout failed with git exit 128; the corrected layout succeeded while revision mismatches and missing checkouts stayed rejected. Current PR checks are completed with successful or skipped conclusions, including the aggregate.

## Impact

Both lane and private-script revision checks remain intact. No time limits, assertions, production behavior, or evidence validation requirements change. The synthetic layout checks do not execute the actual scheduled security suite; integrated main CI and release acceptance remain separate gates.

## Additional Notes

Approved review 5374559393 is bound to the exact head above. Reverting the single-file change restores the previous checkout lookup.
2026-10-01 11:24:37 +08:00
Chris a33dd896e2 fix(ci): restore E2E membership and pagination timeouts (#8281) 2026-10-01 09:00:02 +08:00
AL 6796382cee fix(scanner): validate checkpoints against global cycle fence (#8278)
## Related Issues

Related to rustfs/backlog#2701.

## Summary of Changes

Route scanner checkpoint cycle and leader validation through the global store while retaining the owning set for cache persistence, CAS revisions and publication admission.

## Verification

Two independent final-diff source reviews found no issues across correctness, concurrency and durability, test coverage, compatibility, performance and simplicity on head `8b8fe51d092090b053f552ae283960e2e306be33`. Root approval `5373624714` is bound to that exact head. Regression tests cover real two-pool routing, stale fences, post-save rejection and CAS conflicts; their reported local execution belongs to the PR author, not this merge operation. Current required CI remains pending, and this authorized admin squash does not establish CI or runtime acceptance.

## Impact

Restores checkpoint progress when global cycle and leader state differ from a set-scoped view. No format, retry, timeout, assertion or scanner-policy changes are introduced by this diff. The three prior main scanner failures remain unproved repaired.

## Additional Notes

Full validation must run on the resulting exact main revision. Reverting this patch restores the earlier set-scoped fence lookup and its checkpoint rejection behavior.
2026-10-01 08:52:25 +08:00
overtrue 99a2af8687 docs(release): validate candidates on release branch 2026-10-01 08:37:30 +08:00
Chris 2806a80c91 fix: prevent metadata lock reentry and stabilize CI fixtures (#8257)
* fix(test): establish a writable previous-release upgrade baseline

* fix(ci): reserve capacity for durable admin fixtures

* fix(ci): group durable IAM state fixtures by resource needs

* ci: run E2E doctests with the E2E dependency graph

* fix(ci): separate fixture startup from transport deadlines

* fix(ci): bound pagination after seeding and revisit restored copies

* fix(ci): make filesystem fixture timing deterministic

* fix(ecstore): avoid metadata lock reentry during internal mutations

* test: align recovery fixtures with durable ownership contracts
2026-09-30 20:57:07 +08:00
Chris 8655c38f1d fix(rio): preserve bounded retries after peer EOF (#8272) 2026-09-30 20:18:28 +08:00
Chris 88d03d8199 fix(storage): prevent metadata deadlocks and abandoned writes (#8268)
## Related Issues

N/A

## Summary of Changes

Reject system-bucket incarnation lookups before entering the pool metadata owner. Keep each selected scanner backlog publication cohort in one task so waiter cancellation cannot abandon its remaining serialized conditional writes. Bind repair and replay fixtures to persisted bucket incarnations.

## Verification

Head f6d44603fc received an approval from houseme in review 5365588011. This merge does not add a local runtime validation claim. Full main CI remains a separate publication gate.

## Impact

System metadata writes avoid recursive pool locking. Publication retains per-replica conditional writes and stops a cancelled caller from advancing to its next publication phase. Fixture changes supply the identities required by existing repair admission rules.

## Additional Notes

Reverting this change restores the previous behavior.
2026-09-30 19:49:24 +08:00
Hauser eb350ad20d fix(ecstore): purge empty recursive bucket orphans (#8261)
## Related Issues

Related to rustfs/backlog#2688.

## Summary of Changes

Reclaim metadata-less orphan directory trees after a complete, first-page, empty recursive root listing in an authoritative never-versioned bucket. Both listing implementations use the same fail-closed cleanup decision.

## Verification

Head b4a9120173 received an approval from loverustfs in review 5365460152. This merge does not add a local runtime validation claim. Full main CI remains a separate publication gate.

## Impact

Empty recursive root listings can remove otherwise unaddressable physical residue. Versioned buckets and populated listings remain excluded; unreadable disks, object metadata, unknown files, and uncommitted data prevent cleanup.

## Additional Notes

Reverting this change restores the previous orphan-directory behavior.
2026-09-30 19:24:32 +08:00
KarlEmm e573fb087b ci(helm): sync README to root of helm repository on release (#8263)
Include helm/rustfs/README.md in the uploaded helm-package artifact so
publish-helm-package updates the root README.md in rustfs/helm.

The build-helm-package job copies helm/README.md into helm/rustfs/
before running helm package, but upload-artifact previously uploaded
only helm/rustfs/*.tgz. Because both files share the helm/rustfs/
parent directory, actions/upload-artifact places both the chart
archive and README.md at the artifact root, and download-artifact
extracts README.md into the root of rustfs/helm alongside the tarball.
2026-09-30 15:57:24 +08:00
Chris 099a408d8b test(ecstore): isolate dispatch shutdown recovery fixtures (#8265) 2026-09-30 15:56:48 +08:00
Chris 5c3a2ff277 fix(ci): reserve capacity for profile cancellation probe (#8259) 2026-09-30 15:46:20 +08:00
Chris bb04355a3d fix(heal): retry contended local metadata CAS (#8267) 2026-09-30 15:45:59 +08:00
Chris fab2d7e144 fix(scanner): serialize pause backlog replica publication (#8264) 2026-09-30 15:45:37 +08:00
Hauser d60dfbb826 fix(heal): finalize durable MRF legacy responsibility lifecycle (#8254)
## Related Issues

Resolves the corrected lifecycle fixture findings in PR #8254.

## Summary of Changes

Preserve separate incarnation-bound durable MRF responsibilities and their lifecycle audit records, and align replay fixtures with their checkpoint identities.

## Verification

Two independent source reviews and changed-delta reviews are complete. The final review approved 607a0b9bb0 with no remaining supported findings. Local runtime claims were not independently reproduced for this pull request.

## Impact

Keeps storage generation fences and retained responsibility semantics. Test-capacity reservations preserve existing deadlines and assertions.

## Additional Notes

Squash merge of the currently approved fix under the authorized CI-bypass exception. Main CI and release acceptance remain required.
2026-09-30 14:02:32 +08:00
Chris 6bc2e10bc1 fix(tier): keep recovery worker after startup reload failure (#8260) 2026-09-30 12:05:26 +08:00
Chris 3268c42e00 fix(ci): stabilize registration and rolling upgrade readiness (#8244)
## Related Issues

Follow-up to #8233.

## Summary of Changes

Registration runtime fixtures timed out in the workspace CI lane while the same cases passed in the feature lanes. Reserve nextest capacity for the exact `connect_registration` binary, as already done for related inventory and drive fixtures. Keep its existing deadlines, internal concurrency, assertions and zero-retry policy; report the last watch status on failure.

Rolling upgrades could pass `ListBuckets` readiness while restarted peers still lacked write quorum. Before each mixed-version phase, probe writes through every node outside the asserted workload prefixes. A shared 30-second deadline includes requests and sleeps; only HTTP 503 with `ServiceUnavailable` is retryable, with SDK retries disabled for these probes. The actual compatibility writes, reads, multipart operations and listing assertions remain unchanged.

Add four fast regression tests to the existing PR smoke profile, with matching exclusion from the full profile. No new workflow or job is introduced.

## Verification

- `cargo nextest run --locked --profile ci -p rustfs --lib --test connect_registration --test-threads 4 -E 'binary(/^connect_registration$/) | test(=connect::diagnostics::trace_runtime::tests::local_runtime_rejects_non_private_state)' --no-tests fail --status-level pass --final-status-level fail` passed 23/23 twice in 5.088s and 5.097s: 22 macOS registration tests plus one unrelated control. JUnit intervals confirm capacity reservation; no retries or test-process leaks occurred. The Linux-only inventory case remains for CI.
- `cargo nextest run -p e2e_test --lib --profile ci -E 'test(upgrade_write_readiness_tests)' --no-fail-fast --no-tests fail` passed 4/4 twice in 1.068s and 1.072s, with zero retries. Regressions cover metadata readiness followed by write unavailability, recovery through every writer, immediate permanent-error failure despite an incoming SDK retry configuration, and deadlines for repeated 503s and stalled requests. The original failure is recorded in [the mixed-version upgrade job](https://github.com/rustfs/rustfs/actions/runs/36569700716/job/109415806473).
- `cargo fmt --all --check`, `git diff --check`, `python3 scripts/check_test_wiring.py` and compiled smoke/full membership checks passed. Smoke membership changes from 188 to 192 by adding exactly these four tests; full membership is unchanged. The expected Linux smoke digest was derived from the actual prior Linux listing plus those four platform-independent additions and still requires confirmation by this PR's CI.

Local verification covers the exact source committed in `9b2ea836317ed035a91d1fc9fcf725f70c3098e2` on main `380e98a42cb4fcd0994fed79b30c2c7605deb0bc`. An independent final-diff correctness and reliability review found no findings. Fresh Linux workspace and real mixed-version upgrade runs are required before treating the remediation as fully verified; local fake-target tests do not establish that result.

## Impact

Test scheduling and readiness only; no production behavior, API, dependency, test deadline or compatibility assertion changes. Reserving capacity serializes registration fixture processes within a nextest run. The bounded readiness probes may add startup time while peer write health converges; permanent errors still fail immediately.

## Additional Notes

Rollback by reverting this PR. Existing CI restructuring from #8233 is independent of these follow-up fixes.
2026-09-30 10:06:09 +08:00
Chris 9e33d54269 fix(release): package stable preview tags without updating channels (#8256) 2026-09-30 09:49:27 +08:00
Hauser 498080dec4 fix(storage): default new bucket durability to strict (#8253)
* fix(storage): align new bucket durability with strict defaults

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): defer failure recovery until scan completion

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-30 09:23:32 +08:00
Hauser 296854c3d2 test(heal): cover degraded MRF replay after deletion (#8249)
* test(heal): cover deleted versions during MRF replay

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): exercise degraded MRF replay after delete

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): route quorum errors through test storage API

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-30 09:14:02 +08:00
Jason Kossis 56468fd553 fix(scanner): resume checkpoints across cycle and leader changes (#8252) 2026-09-30 08:18:20 +08:00
Chris 50aa482136 fix(connect): add non-Unix health runtime fallback (#8247) 2026-09-30 08:10:10 +08:00
Chris e870a6d25b fix: separate CPU profile stack lifetime from accumulator (#8250) 2026-09-30 01:36:14 +08:00
Chris 5851d9eb54 fix: filter CPU profile samples to reviewed executable symbols (#8248) 2026-09-30 01:06:24 +08:00
Chris 1e7065101d feat: Capture offline CPU profiles in serving process (#8245)
feat: capture offline CPU profiles in serving process
2026-09-29 22:37:10 +08:00
Chris 380e98a42c fix(test): synchronize top-disk fixture writes with sampling (#8243) 2026-09-29 21:07:45 +08:00
Chris a88225d208 fix(ci): reduce duplicate work and preserve reliable test failures (#8233)
* fix(ci): reduce duplicate work and preserve reliable test failures

* fix(ci): retain protocol evidence and repair stale test fixtures

* test(connect): honor parent deadline during API fixture readiness

* fix(ci): reserve IO capacity for state writer proofs

* test(connect): align RPC fixtures with service capture contracts

* test(connect): cover pinned service capture failures
2026-09-29 21:06:45 +08:00
Chris 1cca0dd25c feat(connect): select offline key for client performance (#8242) 2026-09-29 20:49:59 +08:00
Chris b0a54f0e5c feat(connect): capture offline network performance in server (#8240)
* feat(connect): capture offline network performance in server

* fix(connect): use storage facade in RPC test

* fix(connect): keep RPC test body behind storage facade
2026-09-29 19:41:43 +08:00
Chris 83a2d8b02d feat(connect): capture RPC activity in offline server runtime (#8237) 2026-09-29 18:16:54 +08:00
Chris f661651aad feat(connect): capture API activity in offline server runtime (#8234) 2026-09-29 17:18:26 +08:00
Chris 9298991d14 fix(scanner): restart a finished mixed sweep under its requested plan (#8231)
A bucket sweep that ends mixed clears its position and records the finishing cycle's plan as started. The next cycle requests a new plan whenever the bucket was written in between, so a fresh sweep inherited a stale started plan and ended mixed again. On a continuously written bucket no sweep could ever certify and the census kept the old root.

Start a fresh verification sweep under the requested plan when no durable position remains. Resumed sweeps keep their started plan, so a clean tail still cannot certify an old prefix.

Refs #7108
2026-09-29 16:48:58 +08:00
Chris 3c7d85420c fix(test): control clocks in Connect network fixtures (#8228) 2026-09-29 16:48:34 +08:00
Chris 6a7880c9fc feat(connect): capture offline lock metrics in server (#8230) 2026-09-29 16:29:07 +08:00
hector 0e58bd09d0 fix(ci): run security chain evidence from the lane's own checkout (#8229)
The security workflow checks the repository out into rustfs-repo/ (#7212)
but its chain evidence steps still invoke
scripts/functional_chain_evidence.py relative to the workspace root, which
on the persistent shared runner resolves to a stale checkout left by another
job. The evidence gate then compares that checkout's HEAD against the chain
workflow_sha and rejects the lane before any case runs
("lane checkout differs from chain workflow source").

Run the evidence script from the lane's own checkout so ROOT resolves to
rustfs-repo/, whose HEAD is exactly the chain-pinned workflow_sha.
2026-09-29 16:09:31 +08:00
Hauser fcb60e7322 fix(rpc): preserve walk_dir missing-path errors (#8226)
## Related Issues

rustfs/rustfs#8217 and rustfs/backlog#2683.

## Summary of Changes

Reviewed 539ed275db against d2175d1e1e. No findings. Capable callers requesting missing-path reporting recover explicit pre-output FileNotFound or VolumeNotFound errors; partial-output and other failures continue to fail the stream.

## Verification

Two independent source reviews completed across all lenses below; the frozen diff matches Git and passes diff whitespace checks. Traced the authenticated handler, bounded first-chunk preflight, cancellation ownership, HTTP status/token classification and quorum error handling. The new tests cover ordinary streaming, typed missing errors, byte preservation and mixed missing/I/O failures.

Author-reported focused tests and Docker reproduction were not rerun or independently audited in this review. The reported Docker comparison predates the final report_notfound-only gate; its scope is not distributed acceptance of the final head.

## Impact

Correctness: no findings; missing errors remain typed only before output. Security/trust: no findings; authentication, body digest and operation/status/token restrictions remain. Compatibility: no findings; legacy clean EOF remains, with the documented old-server limitation. Concurrency/durability: no findings; receiver ownership cancels an abandoned producer. Simplicity: no findings. Coverage: no blocking gap identified. Performance: no findings; preflight retains one bounded chunk and ordinary report_notfound=false streams start immediately.

## Additional Notes

This review does not establish that the patch fixes any current main CI failure. Full main CI and release acceptance remain separate gates.
2026-09-29 15:29:59 +08:00
Hauser 0a61f4a81e fix(heal): park healthy legacy MRF intents (#8225)
## Related Issues

rustfs/backlog#2682 and rustfs/rustfs#8192.

## Summary of Changes

Reviewed ca970f55ec against d2175d1e1e. No blocking code findings. The hold requires a completed healthy legacy check and an exact incarnation, lease, object, version and scope; it preserves the durable journal and rechecks on restart.

## Verification

Two independent source reviews completed, covering the lenses below. The frozen diff matches Git and passes diff whitespace checks. Reported Cargo and Docker results were not rerun or independently audited in this review.

One factual correction to the PR description: `legacy_sigkill_replay_repairs_without_releasing_unverified_responsibility` uses the default four-member fixture, with three replicas before restoration and four afterward (`mrf_partial_write_test.rs:559,567`). That named regression exercises the production manager path, but is not EC12+4. Other tests in the file use sixteen disks.

## Impact

Correctness: no findings; proofless legacy health never becomes a verified receipt. Security/trust: no findings; exact identity fences remain. Compatibility: no findings; journal encoding remains unchanged. Concurrency/durability: no findings; replay retains one checkpointed owner and replacement generations become retryable. Simplicity: no findings. Coverage: no blocking gap identified, with the test-scope correction above. Performance: no findings; held entries leave the retry index without adding per-admission queue scans.

## Additional Notes

The current main journal failure concerns DecodeFailure, which follows the separate ECDecode task path. This review does not establish that this PR fixes that failure or the scanner-cycle failure, and does not establish a passing main CI or release gate.
2026-09-29 15:28:46 +08:00
Chris d2175d1e1e fix(test): keep bucket disk faults across reconnects (#8224) 2026-09-29 14:07:30 +08:00
Chris 2019715d1a Fix fresh capacity probes for formatted local disks (#8222) 2026-09-29 12:23:06 +08:00
Chris 31f05a44af fix(test): use expect_err for offline state root rejection (#8220) 2026-09-29 11:22:16 +08:00
Chris a1724f3dbe feat(connect): capture signed offline service health (#8219) 2026-09-29 10:50:23 +08:00
Chris c62979a45b fix(connect): secure new offline state roots (#8218) 2026-09-29 10:13:58 +08:00
Chris e14eacc5ed fix(rpc): keep legacy format reads in their original namespace (#8216) 2026-09-29 09:24:39 +08:00
Chris 47af5565c7 fix(ci): preserve cluster startup failure evidence (#8213) 2026-09-29 08:50:09 +08:00
Chris f0ce628a7f feat(connect): capture offline native threads in service process (#8215) 2026-09-29 08:48:56 +08:00
Chris 32cc1eb76c Capture offline memory profile from the running service (#8214)
feat(connect): capture offline memory from the running service
2026-09-29 08:43:25 +08:00
Chris 84fe13989e fix(ecstore): retain read quota and control reserve test hedging (#8211) 2026-09-29 07:16:31 +08:00
Chris c8f42b8dca fix(ci): isolate scanner deadline and expose test failure details (#8210)
* fix(ci): isolate scanner deadline fixture and expose readiness errors

* test(ecstore): report unexpected capacity reservation errors
2026-09-29 07:16:17 +08:00
Chris c9acf01fd0 fix(tier): reread mutation intents after lock contention (#8209) 2026-09-29 07:16:01 +08:00
Chris 87e9a84a4e fix(ecstore): preserve RPC status when cloning storage errors (#8207)
* fix(ecstore): preserve RPC status when cloning storage errors

* test(heal): retain clone coverage without redundant ownership
2026-09-29 07:15:46 +08:00
Chris af0ae798f8 fix(test): serialize replacement_bucket_metadata heal test under parallel load (#8212) 2026-09-29 07:07:31 +08:00
dependabot[bot] a2bdcf497b build(deps): bump the dependencies group with 14 updates (#8176) 2026-09-29 01:59:07 +08:00
Chris 5564932f38 fix(heal): retry cancelled internode RPC operations (#8205) 2026-09-29 01:58:35 +08:00
Chris 8778d55e46 fix(ci): preserve diagnostics for rio storage test failures (#8204) 2026-09-29 00:15:26 +08:00
Chris 5ebc1e480b test: stabilize heal timing and improve heartbeat replay coverage (#8150)
fix: preserve heartbeat replay and stabilize timing tests
2026-09-28 22:07:15 +08:00
Chris 4d782141d8 fix(test): add missing unavailable_drives field to readiness test initializers (#8200)
Commit 151103a609 added the  field to
StorageReadinessDetails but did not update two test initializers in
readiness.rs, causing E0063 compile errors in --all-targets builds.
2026-09-28 21:26:42 +08:00
Hauser 3bc36acf7f fix(notify): persist Docker queue stores and expose open errors (#8201)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-28 21:24:12 +08:00
Chris 4a91c75152 fix(ci): restore checks and multipart migration retries (#8196)
* fix(ci): restore multipart lint and full E2E membership checks

* fix(ecstore): preserve multipart completion identity during migration
2026-09-28 21:22:49 +08:00
Chris 2648776be7 fix: accept catalog files in health service artifact (#8198)
* fix: accept catalog files in health service artifact

* ci: retain configured health acceptance runner
2026-09-28 20:45:26 +08:00
RustFS 9442e89f5f fix(iam): require an explicit permission for force-delete (#8154)
* fix(iam): require an explicit permission for force-delete

A force-delete header no longer inherits s3:* or consoleAdmin. Bucket
force-delete requires s3:ForceDeleteBucket whenever the header is present,
and recursive object force-delete requires s3:ForceDeleteObject. A plain
delete keeps the existing checks.

Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* test(e2e): keep force-delete header names static

The bucket force-delete helper must pass a static header name into the
SDK request mutator.

Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* test(e2e): move the force-delete header into the request mutator

The SDK request customizer requires a static header name owned by the
closure.

Co-authored-by: RustFS <hello@rustfs.com>
Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>

* fix(iam): keep force-delete out of NotAction grants

NotAction now uses plain wildcard matching, so NotAction "s3:*" still
excludes force-delete. An Allow statement grants s3:ForceDeleteObject or
s3:ForceDeleteBucket only when its Action list names the action; a
NotAction-only Allow never does. The rule applies to both IAM and bucket
policy statements.

Also build the invalid-header errors with S3Error::with_message to keep
the s3s footprint at its baseline, and fix a clippy single_match.

---------

Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-09-28 19:08:33 +08:00
RustFS fc5609bbb0 fix(s3): make retried CompleteMultipartUpload idempotent (#8153)
Record the upload id on the completed object so a lost-response retry
returns that object's ETag instead of NoSuchUpload, while a different
part list or a replaced object keeps the existing errors.

Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-28 19:08:10 +08:00
Chris 64bd224b3b fix(connect): accept the health-era heartbeat and measured Top windows (#8195)
Two Connect tests fail on main in every Test and Lint lane.

Heartbeat: #8155 replaced "today's capabilities minus
profile.memory.service@1" with "the pre-health set minus it". That drops
the release that advertised health.check.service@1 but not yet in-service
memory profiling, so a pending heartbeat persisted by that release now
fails validation and the runtime stops. Accept that release as its own
frozen list.

Top disk/net: execute_*_job sets max_duration_millis to the requested
duration, then the capture compared the measured sleep against it. The
timer only overshoots, so a job that ran exactly as requested was
rejected with LimitExceeded whenever the overshoot reached 1 ms. Report
the authorized window, as Top API already does; validate_capture bounds
it by the limit.
2026-09-28 19:07:39 +08:00
Chris 71f8b607ad fix(ci): restore mainline and scheduled test reliability (#8187)
* fix(ci): restore mainline and scheduled test reliability

* fix(ci): provide GitHub CLI for CPU acceptance

* ci: provide Docker for CPU service acceptance

* ci: provision Python and Docker for OIDC validation

* ci: restore hosted runners for Docker validation

* ci: use verified MinIO release packages for interop

* ci: preserve host ownership of MinIO fixtures

* fix(ci): correct diagnostic limits and isolate startup checks

* test(connect): include object CLI failure details

* test(readiness): initialize unavailable drive diagnostics
2026-09-28 19:07:29 +08:00
GatewayJ cef0c61532 fix(table-catalog): preserve data sequences during file rewrites
Merge the reviewed fix from pull request #8104.
2026-09-28 18:51:58 +08:00
Justin Bradfield fc325bcc56 fix(s3): report x-amz-mp-parts-count on GetObject with partNumber
Merge the reviewed fix from pull request #8115.
2026-09-28 18:51:48 +08:00
cxymds 4a46f5ff3d fix(s3): serve bucket websites on dedicated domains
Merge the reviewed fix from pull request #8183.
2026-09-28 18:48:21 +08:00
cxymds e0a973bf4e fix(s3): preserve prefix listing pagination and marker metadata
Merge the reviewed fix from pull request #8181.
2026-09-28 18:47:03 +08:00
hector 671238458c ci: fail the pool lane when expected step markers are missing
Merge the reviewed fix from pull request #8190.
2026-09-28 18:46:12 +08:00
Chris 151103a609 fix(ecstore): recover remote disks after transient stalls
Merge the approved fix from pull request #8149.
2026-09-28 18:40:30 +08:00
Nikita Bakun cc7d5e3f5f fix(replication): stop sending versionId query once a target rejects it
Merge the approved fix from pull request #8109.
2026-09-28 18:40:20 +08:00
yi111 d556d4dc94 fix(admin): report a missing policy or user as 404 NoSuchResource, not 500 InternalError
Merge the approved fix from pull request #8127.
2026-09-28 18:40:10 +08:00
chapman 6300fbe74f fix: map IAM not-found errors to 404 in InfoCannedPolicy and RemoveUser
Merge the approved fix from pull request #8134.
2026-09-28 18:38:20 +08:00
Chris 6ebc78d115 fix(ci): run profile acceptance on Docker runners (#8191) 2026-09-28 16:14:07 +08:00
Chris 5f4d5e8fe8 fix(ci): run CPU acceptance with Docker
Use the Docker-enabled runner required by the connected CPU service-job test.
2026-09-28 15:38:36 +08:00
Chris e4520f70a1 fix(ci): verify CPU acceptance identity without gh
Use authenticated curl for the pinned release run and artifact checks on sm-standard-2.
2026-09-28 15:13:03 +08:00
hector f54945a4c4 fix(ci): repair pool budgets and Connect test failures (#8155)
* ci: give the pool run a real wall-clock budget

The pool suite budgets up to 24h each for rebalance and decommission
(REBALANCE_TIMEOUT / DECOMMISSION_TIMEOUT in rustfs_pool_expand.sh),
but the workflow step capped it at 45 minutes. Four consecutive runs
died identically: steps 1-7 all PASS, then the rebalance wait was
killed at exactly 45:12 - chain 35680308150, standalone 35702803577,
chain 35757773373, chain 36283120750 - with rebalance at completed=1/4
(~21 minutes in), so a full pass has never been observed.

Make the budget an input (default 240 minutes: covers the observed
rebalance pace plus one decommission pass) and document why. The suite
keeps failing the job through its [POOL-STEP] marker adjudication.

* fix(ci): align pool job budget and timeout contract tests

* fix(connect): preserve legacy heartbeats and isolate I/O tests

---------

Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: RustFS <hello@rustfs.com>
2026-09-28 14:26:21 +08:00
Hauser 4beac12439 fix(connect): return profile capture result directly (#8182)
* fix(connect): return profile capture result directly

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* style(connect): normalize profile capture formatting

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(tests): use async mutex for thread profiles

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(connect): widen diagnostic timing budgets

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-28 14:25:27 +08:00
Chris 6523ab8d9f fix(ci): verify GNU packages with CPU catalogs
Match the archive-member check to the GNU pyroscope catalog packaging condition.
2026-09-28 13:23:46 +08:00
Chris 0609e7ce14 fix(connect): admit complete release CPU symbol catalogs
Bound the release catalog to 32 MiB and 200,000 symbols based on the GNU build measurement; keep complete names and reject catalogs beyond either limit.
2026-09-28 12:09:26 +08:00
Chris afb9ddabb4 fix(helm): route Gateway API traffic to the chart service (#8145)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-28 02:26:56 +00:00
RustFS 2e014d25b3 fix: bound restarted-peer stalls on quorum reads and writes (#8152)
A restarted peer can accept a pooled connection and never send response
headers, so HttpReader::open waited past the client body timeout before
the body-stall timer or erasure hedge could run. Bound that header wait
by the stall timeout and retry the open once on a fresh connection.

After write quorum, MultiWriter still waited out the full disk stall
for a silent peer, which matches the client timeout. Give remaining
writers one second, then drop them so the caller returns.

Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
2026-09-28 02:22:41 +00:00
Chris 719000460b ci(connect): match GNU build runner in acceptance checks (#8180) 2026-09-28 09:08:25 +08:00
hector 528a368144 ci: refresh the preview-release contract for the current build workflow (#8178)
The contract check asserts exact lines in build.yml, and five recent
build-workflow PRs drifted it out of sync, so Security Audit's Workflow
Pin Report job fails for every PR and every push to main since Sep 27
17:34Z (first failure on push run 36337450094). Refresh the assertions
to the current, intentional shapes - contract intent is preserved and
in fact tightened:

- #8160 added PROFILE_ARGS to the cross/native cargo build lines.
- #8171 added 'should_publish == true' as a stricter prefix on the R2
  publication guard and the create-release / upload-release-assets /
  publish-release / update-latest-version conditions.

Verified locally: the full check (including the docker and helm
workflow contract tests) passes against main's files with these
assertions.
2026-09-28 08:54:33 +08:00
Chris d791d7b761 ci(connect): use Node 25 for acceptance workflows (#8179) 2026-09-28 08:52:32 +08:00
Chris 1a251de4f3 feat(connect): execute signed disk diagnostics in the service (#8177) 2026-09-28 08:36:49 +08:00
Chris e634df8611 fix: collect offline Top disk from the running service (#8174)
fix(connect): capture offline disk IO in the running service
2026-09-28 01:35:40 +08:00
Chris f5fbb5f4e1 ci: verify Top disk in the serving process (#8173) 2026-09-28 01:34:29 +08:00
Chris 5d2d539423 ci: default manual builds to artifacts without publishing (#8171)
Default manual builds to artifacts without publishing
2026-09-28 01:10:52 +08:00
Chris acebc9649d feat(connect): capture service runtime profiles over local IPC (#8168)
feat(connect): export service runtime profiles over local IPC
2026-09-28 00:49:45 +08:00
Chris e33542b0c2 fix(scanner): recover after cycle-state persistence failures (#8148)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 16:39:41 +00:00
Chris 2ee0ad1704 Collect bounded metrics from the serving Tokio runtime (#8167)
feat(connect): collect bounded service runtime profiles
2026-09-28 00:20:19 +08:00
Chris 4c5fc3c991 Use pinned release catalogs for Connect CPU acceptance (#8166) 2026-09-28 00:18:41 +08:00
Chris 6d2b1c629e Generate CPU symbol catalogs from release executables (#8165) 2026-09-28 00:12:36 +08:00
RustFS de484b93e0 fix: admit cold reads and readiness at read quorum (#8156)
EC 2+2 still meets read quorum with two of four nodes up. A survivor
that had not cached a bucket returned 503 on GET, HEAD, and List, and
/health/ready left the Service once write quorum was lost. Reads and
Service membership now follow read quorum and shared locks. Writes and
/minio/health/cluster still require write quorum and exclusive locks.

Signed-off-by: loverustfs <155562731+loverustfs@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 13:58:55 +00:00
hector 4f61b007da ci: build linux-aarch64 natively on sm-standard-4-arm and verify every Linux package (#8160)
Route the two linux-aarch64-* release legs to the dedicated arm64 runner
and build them natively (cross: false) instead of cross-compiling via
cargo-zigbuild on x86_64 hosts.

With native legs now executable, extend the packaged-artifact checks that
were previously gated to x86_64-unknown-linux-gnu only:

- Every Linux leg runs rustfs --version and rustfs-cli --help from its own
  package, proving the artifact matches the runner architecture.
- The packaged-console runtime smoke (server boot + console HTTP 200) also
  covers aarch64-unknown-linux-gnu, so both architectures get full runtime
  verification. Musl legs run the binary checks but skip server boot for
  now.
2026-09-27 13:40:23 +00:00
cxymds 1dd354233c test(connect): sync heartbeat capability expectations (#8161) 2026-09-27 12:22:48 +00:00
Chris 6eb14a11d0 fix(ci): run health acceptance on Docker-capable workers (#8158) 2026-09-27 14:26:28 +08:00
Chris 3667eea893 fix(ci): validate health binary without file utility (#8157) 2026-09-27 14:22:49 +08:00
hector e67bbd7bd3 ci: run the chain orchestration jobs on the smoke-testing runner (#8147)
* ci: run the chain orchestration jobs where the gh CLI exists

#8112 moved these jobs to the sm-standard-2 pool, whose images ship
without the gh CLI (a known property of the build fleet, see #7572).
Last night's chain died in prepare before any suite ran:
resolve_functional_candidate.py:34 does subprocess.run(["gh", "api"])
and got FileNotFoundError; complete-chain and the hourly
functional-chain-health canary fail the same way. On Sep 22/23 the same
jobs ran green on GitHub-hosted runners.

Move prepare, complete-chain and the health canary to ubuntu-latest.
The suite lanes stay on smoke-testing.

* ci: run the chain orchestration jobs where the gh CLI exists

#8112 moved these jobs to the sm-standard-2 pool, whose images ship
without the gh CLI (a known property of the build fleet, see #7572).
Last night's chain died in prepare before any suite ran:
resolve_functional_candidate.py:34 does subprocess.run(["gh", "api"])
and got FileNotFoundError; complete-chain and the hourly
functional-chain-health canary fail the same way. On Sep 22/23 the same
jobs ran green on GitHub-hosted runners.

Move prepare, complete-chain and the health canary to ubuntu-latest.
The suite lanes stay on smoke-testing.

* ci: point the chain orchestration jobs at the smoke-testing runner

Per maintainer decision, consolidate them onto the same runner as the
test lanes. Verified on rustfs-smoke-testing as the runner user:
gh 2.45.0, jq 1.7.

* ci: point the chain orchestration jobs at the smoke-testing runner

Per maintainer decision, consolidate them onto the same runner as the
test lanes. Verified on rustfs-smoke-testing as the runner user:
gh 2.45.0, jq 1.7.
2026-09-27 08:26:43 +08:00
Chris 8d033c85d8 fix(ci): install GitHub CLI for Connect health acceptance (#8146) 2026-09-27 08:04:54 +08:00
Chris b7420fe5f2 fix(ci): bootstrap release upload CLIs (#8144)
fix(ci): bootstrap release upload CLIs on self-hosted runners
2026-09-27 08:00:48 +08:00
Chris 5cb1c9e8cb fix(architecture): route health.rs storage imports through storage_api facade (#8143)
The health diagnostics module imported directly from
crate::storage::storage_api, bypassing the architecture migration
guardrail. Add a connect facade module to rustfs/src/storage_api.rs
and redirect the imports. Also fix a clippy::redundant-guards lint
in the coarse-flags match arm.
2026-09-27 07:53:54 +08:00
Hiroaki KAWAI 20356ab709 fix(scanner): build raw enumeration indexes incrementally (#8114)
* fix(scanner): build raw enumeration indexes incrementally

* test(scanner): iterate over restart test entries directly

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 07:49:59 +08:00
Chris 2c5c43b0e0 test(connect): expose typed drive measurement failure context (#8099)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 05:11:47 +08:00
Chris cf096b3c22 fix(connect): pace diagnostic network payload sends (#8096)
* fix(connect): pace diagnostic network payload sends

* test(connect): exercise native network pacing entry

---------

Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-27 05:11:35 +08:00
Chris 82024907c5 test: add health service acceptance workflow (#8140) 2026-09-27 03:55:17 +08:00
Chris e2785846d0 ci: keep gh-dependent release jobs on GitHub-hosted runners (#8137)
The sm-standard-2 fleet has no gh CLI (see #7572), so moving the release publication jobs onto it in #8112 broke every tag release: the 1.0.1-preview.12 build failed in Create GitHub Release with 'gh: command not found'. Move create-release, upload-release-assets, publish-release, cleanup-preview-releases, and package.yml resolve/package back to ubuntu-latest.
2026-09-27 03:37:23 +08:00
Chris 31ac243865 feat(connect): add bounded health service job 2026-09-27 02:50:49 +08:00
Chris 8741bb77ea fix(storage): weight automatic multipart admission by part size (#8118)
* fix(storage): weight automatic multipart admission by part size

* fix(ci): restore filesystem runner capabilities and typos dependency

* fix(ci): make release guard portable and spell out part variables

* ci: restore sm-standard-2 runners for io_uring and distributed e2e jobs

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-26 22:35:39 +08:00
cxymds 212b20f890 fix(ecstore): admit cold bucket metadata at read quorum (#8120)
* fix(ecstore): admit cold bucket metadata at read quorum

Encode a durable creation-commit state in bucket metadata so committed Object Lock buckets can use read quorum for cold loads while pre-physical creation intents remain fail-closed. Drain commit-write fan-out, keep write quorum for uncommitted intents, and add regression coverage for exact, below, migrated, and partial-create quorum boundaries.

* fix(ecstore): fence bucket creation-commit persistence

Address review findings on the creation-commit proof.

Persist the proof only under the bucket metadata transaction fence: at bucket creation, and through a fenced migration that re-reads the authoritative metadata and revalidates physical presence at write quorum before writing. This stops a stale snapshot from reverting an acknowledged configuration update or outliving a delete/recreate.

Establish commitment when Object Lock is enabled on an existing bucket, inside the same configuration mutation, so a healthy cluster no longer rejects object operations with ErasureWriteQuorum.

* fix(ecstore): keep commit fence error message stable

The error(format!) ratchet requires a stable Display for quorum bucketing; use a fixed message instead of embedding the bucket name.

* test(ecstore): reuse canonical Object Lock fixture in regression tests

The s3s footprint ratchet is shrink-only. Use the existing ENABLED_OBJECT_LOCK_CONFIG static instead of naming s3s DTO types in store tests.

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-26 21:40:40 +08:00
hector 7d3d0b6c2b ci: align the matrix contract with the 36-combination sweep (#8136)
Companion to rustfs/auto-testing#109: the 4x1 topology is removed from
MATRIX_TOPOS because the product rejects a 1-drive-per-node distributed
pool at startup (FATAL BelowMinimum). The unfiltered sweep is now 36
combinations / 300 cases; update the contract totals and comments so an
unfiltered sweep is still held to an exact expected case count.
2026-09-26 20:31:32 +08:00
hector d317bb3274 ci: run the KMS Vault lanes on ubuntu-latest again (#8133)
#8112 replaced ubuntu-latest with sm-standard-2 across workflows. Both nightly
KMS Vault lanes need a working Docker daemon; the sm-standard-2 pool does not
provide one, so both lanes die in seconds on 'Docker is not available' while
the package compiles and publishes fine - and the functional chain never fires
because its gate requires a successful build run. Restore the lanes to
ubuntu-latest, as documented in the lane comment before #8112.
2026-09-26 19:12:18 +08:00
Hauser 5cac685e9a feat(ecstore): integrate io_uring backend improvements (#8105) 2026-09-26 17:14:17 +08:00
Hauser d8bf268885 fix(scanner): bound SNSD deep scans and checkpoint cloning (#8126)
* fix(scanner): bound SNSD deep scans and checkpoint cloning

* ci: run mount-dependent jobs on hosted VMs

---------

Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
2026-09-26 06:06:50 +00:00
hector fdd753a1dd ci: run distributed e2e on ubuntu-latest to restore tmpfs mounts (#8128) 2026-09-26 12:00:26 +08:00
Chris de80e8a673 fix(ci): make shared checks portable across runners (#8124)
* fix(ci): isolate monitor argument checks from runner tools

* fix(ci): install the Typos action download dependency

* fix(ci): make release policy matching portable across awk variants
2026-09-26 10:55:06 +08:00
RustFS 4cf45e9ed2 Update bug report storage requirements: allow SAN/JBOD, recommend XFS (#8125)
docs(github): clarify supported storage in bug reports

State that local disks, SAN/Fibre Channel volumes, and JBOD are supported,
recommend XFS, and keep the existing unsupported remote/shared filesystem rule.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2026-09-26 10:43:33 +08:00
Chris 9f8acf688c feat(github): add issue forms with storage support checks (#8123) 2026-09-26 10:07:49 +08:00
Chris 96fc26cd6a fix(release): defer installation updates until artifacts are live (#8122) 2026-09-26 09:17:15 +08:00
hector edcc81a8fd ci: replace ubuntu-latest runner with sm-standard-2 across workflows (#8112)
* ci: replace ubuntu-latest runner with sm-standard-2 across workflows

* ci: keep scheduled-validation monitors on hosted runners

The freshness and watchdog jobs report stalled scheduled validations.
Running them on the same sm-standard-2 pool means a pool outage stalls
the monitors too, so nothing reports it.

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-09-25 10:02:45 +08:00
Hauser 286bbb8e1f chore: upgrade workspace dependency releases (#8107)
* chore: upgrade workspace dependencies

* chore: refresh dependency lockfile

* fix(rebalance): avoid waiting on active entry gate during stop preparation
2026-09-24 21:24:14 +08:00
cxymds 0913686b12 fix(heal): detect stale delete-marker metadata (#8102) 2026-09-24 11:16:02 +08:00
George Melikov df64242f64 fix(ecstore): resume scan_dir past a dash-suffixed sibling directory (#8071)
A flat (recursive) listing resumed inside "s/" re-emitted everything in
its sibling "s-x/". scan_dir dropped the entries before forward_to by
comparing directory names without their trailing slash, where "s" sorts
before "s-x", while the keys they stand for sort the other way round:
'-' (0x2d) is below '/' (0x2f), so all of "s-x/..." precedes "s/...".
The drain stopped at "s" and kept "s-x".

The next page then started with keys at or before the marker; the
listing layer filtered all of them out, found no more candidates and
answered IsTruncated=false. A bucket of 104,137 objects with backup
directories named "<id>" and "<id>-rollbacks" listed as 5,000; two such
directories of 1,200 keys each listed as 2,000.

Compare every remaining entry as the key prefix it stands for, slash
included, and keep it only when it sorts at or after forward_to or
contains it. The remainder of forward_to is taken before `current` is
trimmed, so this also holds below the bucket root; retain also drops
entries when all of them precede forward_to, which the drain never did.
2026-09-24 07:13:30 +08:00
Chris d3b75e695e test(connect): add scheduler receipt acceptance workflow (#8078)
* test(connect): add scheduler receipt acceptance workflow

* chore(deps): upgrade crates and fix faster-hex advisory (#8098)

* test(connect): isolate scheduler acceptance workflow

---------

Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-24 03:34:27 +08:00
Chris dde1aa77a1 fix(connect): preserve diagnostic schedules across restart (#8092) 2026-09-24 03:28:37 +08:00
Chris be01e513be feat(connect): sample memory within the running service (#8091)
* feat(connect): sample memory within the running service

* test(connect): add official memory service acceptance

* test(connect): consume the final memory job without cloning
2026-09-24 03:25:28 +08:00
cxymds 9ab034df62 fix(heal): preserve retryable writer failures (#8089)
Co-authored-by: Hauser <housemecn@gmail.com>
2026-09-24 02:41:16 +08:00
Tyler Hillery 624463c5ba fix(s3): default CopyObject content-type to binary/octet-stream on REPLACE (#8073)
fix: default CopyObject content-type to binary/octet-stream on REPLACE
2026-09-23 18:04:05 +00:00
Chris 713920dcd6 docs: update license copyright to RustFS, Inc. (#8097) 2026-09-24 00:56:08 +08:00
GatewayJ f219a0aba8 perf(ecstore): parallelize multipart I/O setup and metadata reads (#8085)
* perf(ecstore): parallelize multipart I/O setup and metadata reads

* perf(ecstore): share multipart paths and increase read concurrency

* fix(ecstore): route multipart benchmark through storage API facade
2026-09-23 22:19:23 +08:00
Hauser 311e4306e6 chore(ci): complete action pin and HAProxy upgrades (#8090) 2026-09-23 19:20:39 +08:00
Hauser 7f0f4941cf chore(ci): refresh checkout pin in validated workflows (#8083)
* chore(ci): update checkout pin in validated workflows

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): pin migrated checkout alerts to requested commit

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(ci): pin additional checkout workflows to verified commit (#8084)

* chore(ci): pin more checkout workflows to requested commit

Update six additional workflows to the verified upstream checkout commit without changing their permissions, inputs, or triggers.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(ci): pin functional and OIDC checkout uses

Extend the verified checkout commit pin to functional-chain and OIDC workflows without changing their behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-09-23 09:27:54 +00:00
cxymds 655bb470b8 fix(heal): retry faulty storage disk errors (#8086) 2026-09-23 07:48:39 +00:00
cxymds 56a4099f99 fix(heal): recreate missing bucket volumes (#8082)
* fix(heal): recreate missing bucket volumes

* fix(heal): use typed error for empty targets
2026-09-23 14:51:24 +08:00
hector ee6de7d786 ci(package): build gnu and musl DEB/RPM variants with distinct file names (#8079)
* ci(package): build gnu and musl DEB/RPM variants with distinct file names

The Build and Release workflow produces four Linux binaries
(x86_64-gnu, aarch64-gnu, x86_64-musl, aarch64-musl), but packaging
only consumed the two gnu artifacts. Add matrix entries for the two
musl artifacts so every release ships all four DEB/RPM variants.

The libc variant is now part of the package file names, which would
otherwise collide between gnu and musl builds of the same version:

- deb: rustfs_<version>_<libc>_<arch>.deb
- rpm: rustfs-<libc>-<version>-<release>.<arch>.rpm

The dpkg Package and rpm Name stay plain "rustfs", so gnu and musl
remain mutually exclusive upgrades of one package rather than
co-installable packages fighting over /usr/bin/rustfs.

Dependency declarations now follow the linkage: gnu binaries
dynamically link glibc and keep Depends: libc6 (>= 2.31) /
glibc >= 2.31; musl binaries are statically linked and declare no
libc dependency. The libc variant is also visible in the package
description.

scripts/release/package_versions.sh gains a LIBC argument and its
contract tests cover both variants plus the invalid-libc cases.

* ci(package): align deb/rpm file names with the zip artifact naming

Rename the package file names so every release asset of one build
shares the same stem as its binary artifact, differing only by
extension:

- before: rustfs_<deb_version>_<libc>_<deb_arch>.deb
          rustfs-<libc>-<rpm_version>-<rpm_release>.<rpm_arch>.rpm
- after:  rustfs-linux-<arch>-<libc>-v<version>.deb / .rpm

e.g. rustfs-linux-x86_64-gnu-v1.0.0.zip,
     rustfs-linux-x86_64-gnu-v1.0.0.deb,
     rustfs-linux-x86_64-gnu-v1.0.0.rpm.

Non-development builds embed the raw release tag (with 'v'), like the
zips; development builds embed dev-<full sha>. The dpkg/rpm versions
(including the '~' prerelease ordering) are unchanged - they live in
the package metadata, and a side effect is that release asset names no
longer contain '~' (which GitHub normalizes to '.').

package_versions.sh now takes the target arch (x86_64|aarch64) instead
of the deb/rpm arch pair; the deb Architecture (amd64/arm64) in the
control metadata still comes from the workflow matrix. The two test
workflows that assemble deb download URLs from a release tag
(rustfs-table-test, rustfs-upgrade-test) are updated to the new name,
which also removes their '~'-to-'.' asset name workaround.
2026-09-23 03:17:47 +00:00
1334 changed files with 287017 additions and 41681 deletions
+7
View File
@@ -58,6 +58,13 @@ are repository-relative. Read only the relevant sections during read-only review
- Attach error context once where it is actionable. Do not erase typed errors
below aggregation or quorum layers.
## Rust Boundaries
- Production `unwrap`/`expect` must follow a type guarantee or checked invariant; explain only non-obvious guarantees.
- Public APIs do not return `Result<_, String>`. Library APIs use domain errors unless intentional erasure is part of the boundary contract; expose a stored inner error through `Error::source()`.
- Numeric conversions must not silently truncate. Use fallible conversions for untrusted values; validate finiteness, sign, and range before float-to-integer conversion. Clamp/saturate only when required by the domain.
- Do not add crate-root `#![allow(dead_code)]`.
## Naming
Use Rust API naming: `SCREAMING_SNAKE_CASE` constants/statics, `snake_case`
@@ -2,8 +2,12 @@
- For every changed lock, enumerate overlapping lock sets and construct the
ABBA interleaving. Multiple-lock order must be documented and consistent.
- Mark guard lifetimes and every `.await`, disk, and RPC call inside them.
Estimate contention and timeout behavior under concurrent requests.
- Mark guard lifetimes and every `.await`, disk,
and RPC call; estimate contention/timeouts. Tokio read/write guards across
`.await` need bounded hold time; long-lived reads can wedge writers (#4195).
Keep `std::sync::Mutex` holds brief and never across `.await`.
- Use direct `fetch_*` for unconditional atomic read-modify-write and
`compare_exchange` for conditional updates.
- Object commits remain fenced if the distributed lock is lost after shard
writes and before metadata rename.
- For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent
@@ -15,6 +15,8 @@ Attack the changed behavior, not every subsystem in the repository.
bytes and length.
- For multipart/object commits, fail before/after rename and cleanup; committed
data must remain readable and pre-commit cleanup must not destroy parts.
- Bound recursion over untrusted or persisted input; handle corrupted/cyclic
tree and cache traversals safely.
- For version/index ordering, test `len - 1`, `len`, equal timestamps, missing
versions, and deterministic tie-breaking.
- For directory-object behavior, trace `__XLDIR__` at the store layer; branches
@@ -19,7 +19,8 @@
serialization is required.
- Internal metadata tests assert both RustFS and MinIO keys, not only read-back
through a helper that prefers one key.
- Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
- Avoid duplicate tests of the same production path and poison-value class.
Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
absent vs empty vs nil UUID.
- A focused test proves only the targets/features it builds. Add compilation or
Clippy only for uncovered changed targets.
@@ -1,51 +0,0 @@
---
name: code-change-verification
description: Review a commit, PR, or merged patch when the user requests ordinary code-change verification. Do not combine with adversarial-validation; use that skill instead for explicitly adversarial, substantial, or high-risk RustFS reviews.
---
# Code Change Verification
Use this skill for an ordinary requested review. If the root policy or user calls
for adversarial validation, use `adversarial-validation` instead of running both.
## Core Workflow
### 1) Scope and assumptions
- Derive the change source, target branch, and relevant runtime/version from the
supplied diff and metadata. Ask only when missing context could change the verdict.
- Focus only on requested scope; avoid reviewing unrelated files.
### 2) Risk map
- Prioritize in this order:
- Data correctness and user-visible behavior
- API/contract compatibility
- Security and authz/authn boundaries
- Concurrency and lifecycle correctness
- Performance and resource usage
- Give higher priority to stateful paths, migration logic, defaults, and error handling.
### 3) Evidence-based inspection
- Read each modified hunk with neighboring context.
- Trace call paths and call-site expectations.
- Check for:
- invariant breaks and missing guards
- unchecked assumptions and null/empty/error-path handling
- stale tests, fixtures, and configs
- hidden coupling to shared helpers/constants/features
- Apply root `AGENTS.md`'s finding standard: try to disprove a candidate before
reporting it. Mention an unresolved question only when it could materially
change the verdict; do not fill the report with speculative possibilities.
#### Rust-specific checks
For changed Rust behavior, use the matching sections of [rust-code-quality](../rust-code-quality/SKILL.md). Reuse checks already performed by the selected review workflow. Comment-only or formatting-only Rust diffs do not require the full Rust checklist. Carry its P0–P3 ratings over unchanged and use this skill's output format.
### 4) Findings-first output
- Order supported findings by P0–P3 severity; preserve the Rust ratings above.
Include `path:line`, the failure and impact, a focused fix, and its validation.
- If no supported issues remain, state `No findings` with the reviewed scope and
any material verification limitation. Do not append optional improvements to
make a clean review look productive.
Close after the required review. Recommend additional verification only for an
identified unresolved risk or required gate; reuse evidence for unchanged code.
@@ -1,4 +0,0 @@
interface:
display_name: "Code Change Verification"
short_description: "Prioritize risks and verify code changes before merge."
default_prompt: "Use $code-change-verification for an ordinary requested diff review with prioritized findings."
+7 -106
View File
@@ -1,112 +1,13 @@
---
name: issue-triage
description: Assess whether a GitHub issue is fixed, needs implementation, or can be closed by checking related work and current code. Use for issue completion/triage requests. Status questions are read-only; comment, close, or change labels only when the conversation authorizes that action.
description: Verify issue completion against current implementation and merged work. Use for issue triage; comment, close, or change labels only when authorized.
---
# Issue Triage
Use this skill when the user provides a GitHub issue URL and asks "can this be closed?", "is this already implemented?", "check completion status", or similar triage questions.
- Resolve the issue repository separately from the implementation repository: `rustfs/backlog` tracks work in `rustfs/rustfs`. Pass `--repo` explicitly to GitHub queries.
- Read the issue body, relevant comments, linked PRs, checklist, and sub-issues with `gh issue view`. Search explicit links, the full issue URL, qualified references, and subject keywords; an empty search result does not prove implementation is absent.
- Fetch the implementation base and inspect its current code. For each merged candidate, verify `git merge-base --is-ancestor <merge-commit> <remote>/<base>`; a title, commit message, or merged state alone is insufficient. A local checkout may be stale or on another branch.
- Verify every checklist/sub-issue before declaring completion. For batches, paginate the entire requested issue scope, exclude PR entries, and filter by author only when requested.
- Recommend closing only when all requested behavior is present or evidence shows the issue is superseded. Otherwise name what remains. A status request is read-only; reuse existing authority for comments/closing/labels, and follow [Git and PR rules](../../references/pull-requests.md) before posting. Use Chinese for `rustfs/backlog`, existing repository labels only, and a closing comment naming the verified behavior and PRs.
## Workflow
### 1. Fetch issue context
```bash
gh issue view <N> --repo <owner/repo> --json title,body,state,comments,labels,updatedAt
```
Read the issue body to understand what was requested. Extract:
- The specific feature/fix/behavior described.
- Any linked PRs or commits mentioned in the body or comments.
- Any checklist items or sub-issues.
Resolve the issue repository and implementation repository separately (for example, `rustfs/backlog` tracks work in `rustfs/rustfs`). Pass the implementation repository explicitly to PR queries; the current checkout may belong to another repository.
### 2. Search for related work
Search git history for commits referencing the issue:
```bash
git log --oneline --all --grep="<N>" | head -30
```
Search for related PRs:
```bash
gh pr list --repo <implementation-repo> --search "<issue-url>" --state all --json number,title,state,mergedAt
```
Also search qualified issue references and subject keywords; for same-repository
issues, include `#<N>`. Follow explicit links even without a text match. A search
page with no match does not prove the work is absent.
If the issue mentions specific PRs, check their status:
```bash
gh pr view <PR_N> --repo <implementation-repo> --json state,mergedAt,title,mergeCommit,baseRefName
```
### 3. Verify implementation
Fetch the implementation repository's current base branch. For each merged candidate, verify its merge commit is present and inspect the current code for the claimed behavior; a commit message match alone is not proof:
```bash
git fetch <implementation-remote> <base-branch>
git merge-base --is-ancestor <merge-commit> <implementation-remote>/<base-branch>
```
If the issue describes a specific defect, inspect the fetched base's code rather than assuming the current checkout contains it:
```bash
git show <implementation-remote>/<base-branch>:crates/<relevant>/src/<file>.rs
```
For issues with checklists, verify each item individually. If sub-items are tracked as separate issues, check those too:
```bash
gh issue view <SUB_N> --repo <owner/repo> --json state
```
### 4. Determine verdict
- **All items fixed and merged**: Recommend closing; name the verified PRs and behavior.
- **Some items fixed, some remaining**: Keep open; report each remaining item.
- **Not yet implemented**: Keep open; report what remains.
- **Superseded or no longer relevant**: Recommend closing with evidence.
### 5. Take action
For a status-only request, return the assessment without GitHub writes. If commenting, closing, or label edits are authorized, perform only those actions; do not ask again for authority already given. Prepare the final assessment before asking for any missing authority. Write `rustfs/backlog` issue content in Chinese.
Close with comment:
```bash
gh issue close <N> --repo <owner/repo> --comment "<body>"
```
Comment without closing:
```bash
gh issue comment <N> --repo <owner/repo> --body-file /tmp/triage.md
```
Update labels only when label changes are authorized, using existing repository labels; never add tool-specific labels:
```bash
gh issue edit <N> --repo <owner/repo> --add-label "<existing-label>"
```
Always use `--body-file` for multiline content, never inline `--body`.
### 6. Handle multi-issue batches
When the user asks to check multiple issues (e.g., "check all issues by user X" or "scan backlog for closable issues"):
1. List the full requested scope with pagination (for example `gh api --paginate 'repos/<repo>/issues?state=open&per_page=100'`, excluding entries with `pull_request`). Add an author filter only when the user requested one; the default page/limit is not evidence that all issues were checked.
2. For each issue, run steps 1-5 above.
3. Report a summary table of all triaged issues with verdicts.
## Report
Identify the issue and current state, verified implementation/PR evidence,
remaining items, verdict, and action actually taken. Use a table for batches;
a single issue does not require a heading for each field. Follow step 4's
verdicts without repeating the assessment in another template.
## Notes
- The user may ask in Chinese ("是否可以关闭", "检查完成情况"); respond in the same language.
- When closing, always include a summary of what was fixed and which PRs resolved it — this creates a useful audit trail.
- For issues in `rustfs/backlog`, use `--repo rustfs/backlog`.
- For issues in `rustfs/rustfs`, use `--repo rustfs/rustfs`.
- If the issue has sub-issues (GitHub sub-issues API), check each one's state before declaring the parent complete.
Report the issue, current state, implementation evidence, remaining work, verdict, and action actually taken. Use a table for batches. Write multiline comments via `--body-file`; when closing, post the prepared comment before closing without an inline multiline body.
+13 -86
View File
@@ -1,93 +1,20 @@
---
name: pr-review
description: Review a GitHub PR from a URL or number using its actual base/head and risk-appropriate code review. Use when the user asks for a PR review, not a status lookup or PR wording edit. Publish a review only when authorized; delegation and monitoring follow the requested scope and root AGENTS.md.
description: Review a GitHub PR using its exact base/head and repository risk policy. Use for requested PR reviews; publish or fix only within conversation authorization.
---
# PR Review
Use this skill for PR context and review delivery. An ordinary review request is read-only unless the conversation also authorizes posting or fixes. Reuse that authorization without asking again; prepare the review before requesting any missing publication approval.
1. Resolve the PR repository and remote; the current checkout may belong elsewhere. Read the PR purpose and linked issues, then fetch the actual base and PR head:
## Prerequisites
```bash
gh pr view <N> --repo <owner/repo> --json title,author,state,body,baseRefName,headRefName,baseRefOid,headRefOid
git fetch <repo-remote> <baseRefName> refs/pull/<N>/head
git diff <baseRefOid>...<headRefOid>
```
- Follow root `AGENTS.md`; classify risk with the [review policy](../../references/adversarial-validation.md) and consult relevant [change-style and boundary rules](../../references/implementation.md).
- Select `code-change-verification` for ordinary review or `adversarial-validation` for explicitly adversarial, substantial, or high-risk review; do not run both on the same diff.
## Workflow
### 1. Gather PR context
```bash
gh pr view <N> --repo <owner/repo> --json title,author,state,body,additions,deletions,changedFiles,commits,baseRefName,headRefName,baseRefOid,headRefOid
gh pr diff <N> --repo <owner/repo> --name-only
```
Read the PR body and linked issues to understand the change's purpose. If the PR references an issue, fetch that too:
```bash
gh issue view <ISSUE> --repo <issue-owner/repo> --json title,body,state
```
### 2. Fetch the diff and classify the change
```bash
git fetch <repo-remote> <baseRefName> refs/pull/<N>/head
git diff <baseRefOid>...<headRefOid> --stat
```
Resolve `<repo-remote>` to the PR repository; do not assume the current checkout's `origin` or `main` matches. Record the exact base/head used. If either moved during fetching, refresh the snapshot before reviewing. Classify using the repository review policy; instruction changes that affect agent execution are mechanical, not exempt.
### 3. Review the changed behavior
Group files by functional area to trace callers and invariants. Use the root risk tier's review shape and only matching lenses. File count does not authorize delegation. When delegation is explicitly authorized, high-risk/substantial reviews use exactly two independent reviewers with the applicable lenses split between them; otherwise use two fresh sequential passes. Reviewers do not spawn further agents.
Findings need a concrete failure scenario with `file:line`; a null verdict briefly names the relevant probes. Reuse existing evidence and choose local checks from the final diff under the root verification policy.
### 4. Check CI status
```bash
gh pr checks <N> --repo <owner/repo>
```
Investigate a failed check when it bears on a finding or the user requested CI diagnosis/merge readiness:
```bash
gh run view --repo <owner/repo> --log-failed --job=<JOB_ID>
```
Use current evidence to distinguish pre-existing, flaky, and PR-caused failures. Do not classify them by guesswork or turn a code-only review into unrelated CI repair.
### 5. Synthesize findings
Report the PR, reviewed base/head, and risk tier, then summarize the assessment.
Use the selected review's P0–P3 ratings and root finding standard: supported
findings with `file:line`, failure scenario, and fix, or `No findings`.
State the observed check status, including pending or unavailable checks, and
the verdict (`APPROVE`, `REQUEST_CHANGES`, or `COMMENT`). Do not infer a pass
from missing checks or add style nits to populate a clean review.
### 6. Post the review
Only when posting is authorized, write the review body to a temp file and post via CLI. Refresh the PR head first; if it changed, review the delta and update the verdict before posting:
```bash
# Request changes
gh pr review <N> --repo <owner/repo> --request-changes --body-file /tmp/pr_review.md
# Approve
gh pr review <N> --repo <owner/repo> --approve --body-file /tmp/pr_review.md
# Comment only (no verdict)
gh pr review <N> --repo <owner/repo> --comment --body-file /tmp/pr_review.md
```
For authorized inline comments, use [the submission example](references/posting.md).
Always use `--body-file` or `--input`, never inline multiline `--body`.
### 7. Handle follow-up
Follow the [PR lifecycle](../../references/pull-requests.md) and any explicit monitoring request. For follow-up, fetch the new head and compare the recorded reviewed SHA with the new SHA; revisit affected callers and findings. Never use an unfetched `origin/pull/<N>/head` ref as evidence. Update the posted review or resolve addressed threads only within existing authorization.
## Notes
- The user may ask for review in Chinese; respond in the same language but keep the review body in English per AGENTS.md rules.
- When the user asks for "多角色对抗 review", run the full adversarial validation protocol — this skill's step 3 covers that.
- If the PR is from a fork, check `maintainerCanModify` before attempting to push fixes.
- For very large PRs, batch the review by functional area while keeping the same bounded review shape.
Record both SHAs; refresh if either moved during fetching. Follow relevant [implementation boundaries](../../references/implementation.md).
2. Apply the [risk policy](../../references/adversarial-validation.md). Ordinary reviews use the root finding standard directly. For explicit adversarial, substantial, or high-risk reviews, use [the domain probes](../adversarial-validation/SKILL.md). File count does not authorize delegation; use exactly two independent reviewers only when authorized, otherwise two fresh sequential passes for that risk tier. Reviewers do not spawn further agents.
3. Check `gh pr checks <N> --repo <owner/repo>`. Investigate failures when relevant to a finding or a requested CI/readiness diagnosis. Distinguish pending, unavailable, pre-existing, flaky, and PR-caused states using evidence; a code review does not authorize unrelated repairs.
4. Report the reviewed SHAs, risk tier, supported findings or `No findings`, material verification limits, check state, and verdict. P0/P1 findings block approval. Severity: P0 = demonstrated data loss, security breach, remote crash, or deadlock; P1 = correctness, compatibility, or material hot-path regression; P2 = a concrete maintainability defect; P3 = optional style, only when requested.
5. When posting is authorized, follow [Git and PR rules](../../references/pull-requests.md), recheck the remote head, and review any delta before submitting. Use `gh pr review` with `--approve`, `--request-changes`, or `--comment` and `--body-file`; use [the API example](references/posting.md) for authorized inline comments. Reuse existing authorization.
6. Follow the repository PR lifecycle and explicit monitoring scope. Fetch new heads before comparing to the reviewed SHA; never rely on an unfetched `origin/pull/<N>/head`. Push fixes or resolve threads only when authorized; check `maintainerCanModify` before a fork push.
-116
View File
@@ -1,116 +0,0 @@
---
name: rust-code-quality
description: Run a focused Rust quality review when the user requests one or a selected review workflow needs Rust-specific checks for changed behavior. Do not auto-load for every implementation edit, comment-only or formatting-only Rust diff, or repeat an already completed review.
---
# Rust Code Quality Gate
Use this skill for a dedicated Rust review to cover rules that `cargo clippy`
does not catch.
Search matches and checklist items are candidates, not findings. Apply the root
finding standard; distinguish a demonstrated bug, an explicit rule violation,
and an optional preference. P2/P3 suggestions do not need to be invented or
included in an otherwise clean correctness review.
## Quick Start
1. Identify changed `.rs` files.
2. Run the matching candidate searches on changed files.
3. Apply the manual checklist sections whose behavior the diff touches.
4. Report or rebut every finding with evidence; P0/P1 findings block approval.
Fix them when implementation is authorized; a read-only review reports them.
## Automated Checks
Use these searches to find candidates in changed `.rs` files. Inspect syntax,
`#[cfg(test)]` scope, and the changed hunk before reporting a finding; text
filters do not reliably distinguish production code from tests.
```bash
# 1. unwrap/expect candidates
rg -n '\.unwrap\(\)|\.expect\(' <changed-files>
# 2. Silent type truncation via `as` cast
rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
# 3. String as error type
rg -n 'Result<.*String>' <changed-files>
# 4. Box<dyn Error> in public APIs
rg -n 'Box<dyn.*Error' <changed-files>
# 5. println/eprintln in production
rg -n 'println!|eprintln!' <changed-files>
# 6. Ordering::Relaxed usage (verify each is intentional)
rg -n 'Ordering::Relaxed' <changed-files>
# 7. Default substituted for a possibly-required value (judge each: is the value optional by domain?)
rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
```
## Manual Review Checklist
For the Rust diff under review, verify:
### Error Handling
- [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment
- [ ] No `Result<_, String>` in public API signatures
- [ ] Public library APIs use domain errors unless deliberate error erasure at a boundary is part of the contract
- [ ] `Error::source()` is overridden when inner error is stored
- [ ] Error messages are actionable without exposing secret input
### Type Safety
- [ ] No silent `as` truncation (negative→unsigned, large→small)
- [ ] Fallible numeric conversions use `TryFrom`/`try_into()` and return a typed error; clamp or saturate only when the domain explicitly requires it
- [ ] Floating-point to integer conversion validates finiteness, sign, and range before conversion
### Concurrency
- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
- [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
- [ ] Atomic read-modify-write uses the direct `fetch_*` operation when possible; use `compare_exchange` only for conditional updates
- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
### Memory and Performance
- [ ] On an identified hot path, report cloning or allocation only with a concrete per-request/per-object cost or benchmark signal
- [ ] Prefer borrowing, moving, `Bytes`/`Arc`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs
### Recursion Safety
- [ ] Recursion over untrusted, persisted, or otherwise unbounded input has a depth limit or uses iterative traversal
- [ ] Tree/cache traversals handle corrupted/cyclic input safely
### Testing
- [ ] Tests have an observable failure criterion; delegated assertions, `#[should_panic]`, snapshot/property checks, and meaningful `Result` failures do not need a redundant `assert!`
- [ ] Use `expect` only when its message improves failure diagnosis; do not add boilerplate to self-evident test setup
- [ ] Test volume and line count are never treated as production-code growth
### Serde
- [ ] Structs from untrusted input reject unknown fields where the compatibility contract permits; otherwise validate security-critical fields explicitly and test the supported input shape
- [ ] `#[serde(default)]` not used on security-critical fields without validation
### Code Hygiene
- [ ] No `#![allow(dead_code)]` at crate root
- [ ] No camelCase statics or Hungarian notation
- [ ] New string literals don't duplicate existing constants
### Reuse and Necessity
- [ ] No new helper duplicates `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, a relevant direct dependency, or plain std/tokio behavior; reused helpers match the call site's semantics
- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
- [ ] Comments avoid narration and change history while completely stating non-obvious lock, `SAFETY`, durability, compatibility, and unwrap invariants
- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
## Severity Classification
- **P0 (Block merge)**: demonstrated data loss, security breach, remote crash, or deadlock
- **P1 (Must fix)**: concrete correctness, compatibility, or material hot-path regression
- **P2 (Should fix)**: avoidable duplication or maintainability issue with a concrete simpler replacement
- **P3 (Nice to fix)**: local style or clarity issue with no behavioral risk
## Output Template
Use the calling review's output format. For a standalone review, report supported
findings with severity, location, impact, fix, and validation, or `No findings`.
Include only material unverified checks. Candidate counts are not a quality
metric and do not need a separate scan report.
@@ -1,34 +0,0 @@
---
name: rustfs-logging-governance
description: Add or review RustFS `tracing` events with the repository field shape, level policy, privacy boundaries, and guardrails. Use when a change adds or edits a tracing macro/instrumentation site or the logging guardrail script.
---
# RustFS Logging Governance
Apply this skill only to changed logging sites; do not turn a local log edit into
a broad logging cleanup.
## Workflow
1. Read the changed function/module context and classify the site as lifecycle,
request/hot path, fallback, external fetch, or summary.
2. Match neighboring structured events and reuse existing `EVENT_*`,
`LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*` constants.
3. Put stable fields first (`event`, `component`, `subsystem`, `state`/`result`,
then context) and a short label last.
4. Select the level by operational meaning:
- `error`: behavior/security-affecting failure;
- `warn`: degraded/fallback/operator-actionable state;
- `info`: low-frequency lifecycle/mode change;
- `debug`: targeted diagnostics;
- `trace`: repetitive request/object/shard success paths.
5. Never log secrets, tokens, auth headers, credential payloads, raw
attacker-controlled bodies, or merged config dumps. Error strings and
`Debug` output are log surfaces too.
6. Prefer one aggregate summary over inventories or startup banners.
7. Run `./scripts/check_logging_guardrails.sh` and the checks selected by root
`AGENTS.md`.
Read [logging-governance.md](references/logging-governance.md) only for a broad
logging audit, event-model migration, or guardrail expansion. Ordinary single-
site edits do not require the full workspace scope map.
@@ -1,4 +0,0 @@
interface:
display_name: "RustFS Logging Governance"
short_description: "Standardize RustFS logs with structured events and guardrails."
default_prompt: "Use $rustfs-logging-governance to standardize or review RustFS logging, reduce noise, and update guardrails."
+27 -141
View File
@@ -1,156 +1,42 @@
---
name: rustfs-release-publish
description: "Run the end-to-end RustFS console gate, version bump, preview validation, human confirmation, and final-tag publication pipeline. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
description: Publish a RustFS version through Console, release-branch CI, preview acceptance, human confirmation, and final publication. Use only for an explicit release request.
---
# RustFS Release Publish (preview-validated pipeline)
# RustFS Release
This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (invoked here with the authorized commit/push/PR scope) with a mandatory preview-tag validation loop before the final tag is published.
Require the exact final SemVer target before version edits or publication. If unspecified, inspect current releases/tags and ask with concrete candidates; do not guess the channel or bump. A named bump type resolves to an exact version that still needs confirmation. Report the confirmed target verbatim; resolve any later mismatch before proceeding. Independent read-only preflight may continue.
The binary reports its build tag (`build::TAG` via shadow_rs; `SHORT_VERSION` in
`rustfs/src/config/cli.rs`), and `build.yml` derives asset names and preview
classification from that tag. Cargo.toml supplies only the no-tag fallback.
Preview and final tags must therefore share the validated source commit;
their tag-dependent version and asset names differ. The channel and cleanup
constraints are defined once under Preview tag naming and Hard rules below.
## Invariants
Pipeline shape:
- Cargo stores `<target>`, never `-preview.N`. Before publication, installation defaults retain the previous available deliverable. Final and preview binaries embed their own tag names, so version/asset names differ despite identical source.
- Use annotated tags without `v`. Preview names are exactly `<target>-preview.<digits>`; choose the next unused iteration after fetching tags. Preview Releases are prereleases, never Latest; publish only versioned assets/exact Docker tags, never latest assets, R2, Helm, moving Docker channels, or installation/banner updates.
- `release/<target>` owns the candidate. Retain its history, backport only reviewed fixes, and never merge main wholesale or force-push it. Pin `PREVIEW_HASH` to its exact commit with complete passing CI. Preview and final tags both point there; never tag a later main/branch HEAD or insert another bump.
- Phase failures block dependents. Keep preview Releases until final publication and notes generation finish; CI removes Releases while retaining tags. Never move an existing preview or final tag.
```
check console main against its latest Release
-> if ahead: publish console -> wait for Release asset + latest API
-> bump RustFS version files to <target> (final version, ONE commit) -> merge
-> tag <preview-tag> at that commit -> CI green
-> verify preview Release assets -> run binary locally + console checks
-> validate with latest rc client
-> report preview acceptance results -> STOP for explicit human confirmation
-> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
-> CI deletes the <target>-preview.N Releases (tags kept)
```
## Read by Phase
On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
Read only the current phase's procedure; retain completed evidence across turns. Follow root verification, Git/PR, and authorization rules.
## Required inputs
| Phase | Procedure |
|---|---|
| 0 | Clean tree, fetch main/tags, verify GitHub access, then complete the mandatory [Console release gate](references/console-gate.md). A green Console build alone is insufficient. |
| 1–2 | [Prepare/reuse the release candidate and run exact-SHA branch CI](references/release-candidate.md). Source version changes use `rustfs-release-version-bump` with authorized delivery scope. |
| 3 | [Verify the preview build, assets, channels, and notes](references/preview-release.md). |
| 4–5 | [Run the downloaded binary, Console CRUD, and latest-rc matrix](references/preview-acceptance.md). Every check must pass. |
| Confirmation | Complete the gate below and end the turn. |
| 6 | After fresh confirmation, [publish and verify the final tag](references/final-release.md) at `PREVIEW_HASH`. |
| 7 | [Reconcile Security Advisories, milestones, installation references, and the website](references/post-release-updates.md) only after complete publication. |
- Final target version, for example `1.0.0-beta.10`.
- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
## Human Confirmation
If the target version is missing or ambiguous, collect the current release/tag baseline and ask before version edits or publication. Continue independent read-only preflight while the answer is pending (see the semver gate below).
After Phases 3–5 pass, recheck that the release branch still equals `PREVIEW_HASH`. Report the target, branch, preview tag/hash, branch CI run, preview Release URL, Console result, and rc matrix. Explicitly ask whether to publish the final tag and end the turn without creating/pushing it.
## Semver gate — resolve the target before version edits or publication
Only a new affirmative user reply to that exact report authorizes Phase 6. The original release request, earlier confirmation, silence, or automated follow-up does not count. Confirmation binds the target, preview tag, and hash; a changed value or repeated/failed acceptance cycle invalidates it. Revalidate and obtain fresh confirmation before continuing.
Versions follow [SemVer 2.0.0](https://semver.org/). Precedence reminder:
## Recovery
```
1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
```
- Before the final tag exists, land fixes on main, backport to the candidate, verify target versions, and restart branch CI/acceptance. Use a new preview iteration once a preview was tagged; main may advance independently.
- Once the final tag exists, retry publication jobs for that exact tag. A source fix requires a newly confirmed target, not retagging or a newer preview. Phase 7 retries recheck artifact availability and superseding releases.
- If abandoned after the bump merged, report the unpublished version on main and record whether to revert or let the next release overwrite it. Deleting candidate branches or preview tags requires an explicit decision.
Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
Rules:
- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
## Preview tag naming
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
## Hard rules
- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
- Preview Release assets are versioned and intentionally visible on the Releases page for the duration of validation. Do not label them Latest or use them to update any latest distribution channel.
- Never delete a preview Release by hand before Phase 6 finishes — Phase 4 downloads its assets and the final Release notes are generated while it still exists. Cleanup is CI's job; only step in manually (`gh release delete "<preview-tag>" --yes`, never `--cleanup-tag`) if `cleanup-preview-releases` failed.
- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
- Completing preview acceptance does not authorize the final tag. After Phases 3–5 pass, report the acceptance evidence and stop until the user explicitly confirms continuation. The original release request, an earlier confirmation, silence, or an automated follow-up does not satisfy this gate.
- Confirmation is scoped to the reported `<target>`, `<preview-tag>`, and `PREVIEW_HASH`. A failed or repeated acceptance cycle, including any new preview iteration, invalidates prior confirmation and requires a new one.
- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
## Phase 0 — Preflight
- `git status --short` clean; `git fetch origin main --tags`.
- `gh auth status` works; confirm you can view `gh release list -L 3`.
- Confirm the exact final target version with the user if not explicit.
### Console release gate
Read and complete [the Console gate](references/console-gate.md) before Phase 1. Verify the latest published Console asset and exact commit; if Console main is ahead, complete its release and asset verification first. A successful build alone does not satisfy this gate.
## Phase 1 — Version bump to the final target (once)
- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
- Get the PR merged into main. Record the resulting main commit:
```bash
git fetch origin main
PREVIEW_HASH=$(git rev-parse origin/main) # must contain the bump PR
```
`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
## Phase 2 — Publish the preview tag
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
## Phase 3 — CI and preview Release verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
## Phases 4–5 — Local artifact, Console, and rc acceptance
Read and complete [preview acceptance](references/preview-acceptance.md): verify the downloaded binary's tag/SHA and readiness, exercise Console CRUD with byte-identical download, and pass the full latest-rc command matrix. Any failure blocks final publication. Retain the results for the confirmation gate below.
### Manual confirmation gate
After every Phase 3–5 check passes, report the target, preview tag, `PREVIEW_HASH`, preview Release URL, console result, and rc matrix, then explicitly ask the user whether to publish the final tag. End the turn without creating or pushing `<target>`.
Continue to Phase 6 only after a new user reply explicitly confirms the reported target, preview tag, and commit. A clear affirmative reply to that exact report, such as `确认继续`, is sufficient; if the reply is ambiguous or any reported value changed, ask again.
## Phase 6 — Publish the final tag on the validated commit
No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
git push origin "<target>"
```
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
## Output contract
Always report:
- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
- Manual confirmation gate status (`WAITING_FOR_CONFIRMATION` or `CONFIRMED`) and its exact target, preview tag, and `PREVIEW_HASH`.
- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, the rc command matrix, and the preview-Release cleanup result (deleted Releases plus surviving tags).
- Any deviation from this pipeline and why the user approved it.
Report completed phase results and incomplete work with evidence: Console previous/latest tags, release need, hash and run/Release URLs; target, branch CI/SHA, previews and hash, confirmation state; preview/final assets, latest-channel state, Console/rc results and cleanup; advisory reconciliation results and pending disclosure decisions; both milestone states, installation PR checks/merge state, and website text/link/PR/deployment state. Keep artifact publication separate from follow-up delivery; do not call an open PR a live update. Report deviations and their authorization.
@@ -0,0 +1,22 @@
# Final Release Publication
Read only after fresh user confirmation of the preview acceptance report. The [parent confirmation gate](../SKILL.md#human-confirmation) applies.
## Phase 6 — Publish the final tag on the validated commit
No second source version bump. Before creating the final tag, recheck the remote `release/<target>` HEAD equals `PREVIEW_HASH`. If it moved after confirmation, the approval no longer applies; validate and obtain confirmation for a new preview. The final tag goes on the exact commit the preview validated:
```bash
git fetch origin --tags
git rev-parse "<preview-tag>^{commit}" # must equal PREVIEW_HASH — abort if not
git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
git push origin "<target>"
```
- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
- Verify the preview cleanup: `cleanup-preview-releases` must succeed, `gh release view "<preview-tag>"` must then report `release not found` for every preview iteration of this target, and `git rev-parse "<preview-tag>^{commit}"` must still resolve to `PREVIEW_HASH` (the tag is kept). If the job failed, delete the leftover Releases manually with `gh release delete "<preview-tag>" --yes` and report it.
- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
Never pass `--cleanup-tag` when removing preview Releases. Remove them manually only if `cleanup-preview-releases` failed, after final publication and notes generation. Record deleted Releases and surviving tags.
@@ -0,0 +1,50 @@
# Post-release Security Advisory, Milestone, Installation, and Website Updates
Read only after the final non-preview tag has passed Phase 6. Tag creation or a green binary build alone is insufficient: the release assets, source archive, container images, Helm package, and applicable latest channels must be available. Preview tags never enter this phase.
## Security Advisories
Reconcile advisories after every completed final release, including Console-only releases. Read-only preparation may start earlier; do not claim an unpublished artifact is an available fix.
1. Fetch every page of the repository's live Security Advisories, including triage, draft, published, and closed states. A permission error or partial response blocks reconciliation; it does not mean there are no advisories. Keep unpublished advisory details in private task artifacts, never in a public PR, release note, or issue.
2. For each potentially relevant advisory, map its fix to the exact published tag and verify the regression evidence. Main-branch reachability, a merged private fix, or a green build alone does not prove the shipped version is fixed. Check cherry-picked equivalents and reverted fixes explicitly. For embedded Console or dependency fixes, verify the shipped component version.
3. Determine the first available fixed release and affected version ranges from evidence. Preserve an earlier correct patched version; do not relabel all fixes as the current release. An unchanged server baseline such as a Console-only release requires checking the earlier server release as well. Leave unresolved reports and fixes absent from the tag pending; do not close them merely because a release shipped.
4. Prepare any necessary corrections to package identity, affected ranges, patched versions, fix/release links, and advisory text. Preserve unrelated fields, credit, CVE identifiers, and disclosure coordination. Apply updates only within existing user authorization; read back the result to verify it. Publishing a draft, requesting a CVE, or closing a report is a separate action: require explicit authorization for the concrete advisories and respect any embargo. Release-tag approval alone does not authorize those actions. Prepare the exact proposed changes before asking for missing authorization.
5. Record each relevant advisory's URL, current state, fix evidence, first fixed version, action taken or proposed, and remaining blocker in private task artifacts. Report advisory follow-up separately from artifact publication. A successful release with unresolved advisory work is not complete follow-up; retain that work for the release monitor or a concrete user decision.
## Version milestones
1. Query all pages of both open and closed milestones in the RustFS release repository, matching titles exactly to version strings without a `v` prefix. A failed or incomplete query is not evidence that a milestone is absent; resolve the lookup before making changes.
2. Close the milestone matching the published `<target>` if it is open; an already closed milestone needs no change. If it is missing, report the gap and continue checking the next version. Do not create a substitute for the released milestone or close/reassign its issues or PRs.
3. Use an explicitly planned next version when provided; otherwise, for a stable `X.Y.Z`, use `X.Y.(Z+1)`. For example, after `1.0.1` is published, close milestone `1.0.1` and ensure milestone `1.0.2` exists. For an alpha/beta/rc target with a trailing numeric counter, increment that counter within the same channel unless a different next version was specified. If the next version cannot be determined, ask for that version and report this follow-up as blocked. Creating a planning milestone does not confirm the target of a future release or bypass the semver gate.
4. Reuse an existing next-version milestone, whether open or closed; do not reopen it or create a duplicate. Only when the complete query confirms it is absent, create an open milestone with the exact next-version title. Do not invent a due date or description. After an uncertain creation result, query again before retrying.
5. Read back both milestones and report their titles, URLs, and actual states, including whether the next milestone was reused or created. A failed close, create, or verification leaves milestone maintenance incomplete; retry only the unfinished operation after refreshing live state. Continue independent installation and website work without moving or republishing the release tag.
## Installation references
1. Re-read the published deliverable state before edits or a retry. If a newer release has superseded this target, do not downgrade installation references or the website announcement. Report the superseding release and leave those defaults intact.
2. Invoke `rustfs-release-version-bump` with stage `post-release`, the exact target, and the authorized delivery scope. Prepare the follow-up on current main; do not edit or move either validated release tag. Cargo versions may already be preparing the next target and must remain untouched.
3. Verify the exact Docker image manifest, Release download links, RPM source archive, and Helm repository entry before publishing the installation PR. The chart workflow overrides chart/app versions from the final tag, pins the source checkout to that release, and waits for its default Docker image before packaging.
4. Report the installation PR and its merge state separately from the completed release. Reuse existing merge authorization and repository checks; do not infer new merge authority from this reference.
## rustfs.com top banner
Use the existing announcement configuration from [website PR #112](https://github.com/rustfs/rustfs.com/pull/112). Do not add a second banner or redesign the header.
1. In an isolated checkout of `rustfs/rustfs.com`, read its current `AGENTS.md`, PR template if present, `data/announcement.ts`, and the consuming header component. Search for an existing announcement PR for this target before creating another.
2. Update `homeAnnouncement` in `data/announcement.ts` after the release and installation availability checks pass. Keep `enabled: true`, `badge: 'New'`, and the existing component. For a stable release, use:
```ts
export const homeAnnouncement = {
enabled: true,
badge: 'New',
message: 'RustFS <target> is now available! Please',
linkText: 'upgrade and try',
href: '/download',
} as const
```
Replace `<target>` with the exact published version. Reserve GA wording for the actual first stable release. For a final alpha/beta/rc tag, explicitly call it a prerelease; use its GitHub Release URL if `/download` does not expose that target. Never announce a `-preview.N` build here.
3. Use the site's current package manager and required checks. For this config-only change, type-check and lint the changed file, build the site, and inspect the generated homepage for the version, visible link text, and destination. Confirm `/download` actually offers the announced version before using that link.
4. Commit, push, and create or update the website PR when included in the authorized release scope. Include the published RustFS Release URL and checks run. Keep merge and production-deployment authority separate; reuse permission already given, but this reference grants none by itself.
5. After an authorized merge/deployment, verify the live `https://rustfs.com/` banner and its destination. If only the PR is ready, report that state and URL; do not report the banner as deployed. A website failure does not invalidate or move the published release tag: retry this phase after resolving the failure.
@@ -0,0 +1,24 @@
# Preview Release Verification
Read for Phase 3 after the candidate passed exact-SHA CI. The [parent release invariants](../SKILL.md) apply.
## Preview channels
- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Helm, or moving Docker channel tag is a pipeline failure. Docker images may use the exact preview version tag, including its existing variant suffixes.
- Preview Releases are cleaned up by the `cleanup-preview-releases` job after `publish-release` succeeds for the deliverable tag. It deletes every Release whose tag is exactly `<target>-preview.<digits>` and never passes `--cleanup-tag`, so the tags survive.
## Phase 3 — CI and preview Release verification
- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
- Confirm the preview tag resolves to the Phase 2 CI-validated `PREVIEW_HASH`. If the release branch advances during acceptance, repeat Phases 2–5 with a new preview tag.
- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
- Confirm Helm is skipped. If preview Docker images are published, confirm they use only exact preview tags (and variant suffixes), with no `latest`, `alpha`, `beta`, or `rc` channel updates. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; it does not authorize advancing any installation default.
## Release notes
- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag — cleanup runs after the notes are generated, so the preview Release is still present and would otherwise be picked as the baseline. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
@@ -0,0 +1,37 @@
# Release Candidate Preparation
Read for Phases 1–2. The [parent release invariants](../SKILL.md) and Console gate apply.
## Candidate Resume Preflight
- `git status --short` clean; `git fetch origin main --tags`.
- `gh auth status` works; confirm you can view `gh release list -L 3`.
- Confirm the exact final target version with the user if not explicit.
- Check whether `release/<target>` already exists before creating it; a resumed release reuses the existing branch and its history. For an in-flight release that already has a preview tag but no branch, anchor the new branch to that tag's commit only after verifying its target versions and prior acceptance evidence.
## Phase 1 — Source version bump to the final target (once)
- If `release/<target>` exists, verify its Cargo.toml and workspace members in Cargo.lock read `<target>` and skip the source bump, even if main has advanced. If the branch versions differ, stop and resolve the mismatch without downgrading main.
- If no branch exists but an unfinished preview tag does, verify its commit has `<target>` in both version files and skip the bump; Phase 2 anchors the branch to that commit. Stop if those files disagree. Otherwise, if main already has `<target>` in both files, skip the bump. Installation references intentionally retain the previous published deliverable during preview; if they were advanced to an unavailable target, restore that baseline before choosing the next candidate.
- Only when neither the release branch, an unfinished preview tag, nor current main already has `<target>`: if main is at a later version, select and verify a historical commit containing `<target>` rather than downgrading main; block if none exists. Otherwise invoke the `rustfs-release-version-bump` skill with stage `prepare`, the final `<target>` (NOT a preview version), and full GitHub flow (commit/push/PR).
- If a bump PR was needed, get it merged into main and record its merge commit. For a new release branch when main already has `<target>`, identify the main commit that introduced those versions. Phase 2 chooses a release candidate containing that commit; an existing release branch keeps its own history and a newer main HEAD is not required.
Do not wait for the moving latest main HEAD to become green. The release-branch CI in Phase 2 validates the chosen commit before a preview tag is created.
## Phase 2 — Pin and validate the release candidate, then publish the preview tag
- Use `release/<target>`. On the first attempt, create it from a selected main commit containing the version bump and intended changes; it need not be the latest main HEAD. On a restart, retain the existing branch and backport only reviewed fixes from main. Verify the branch's Cargo versions still equal `<target>`. If adopting an existing preview, run this branch CI before continuing its acceptance; reuse the tag only when it points to the validated SHA.
- The current `ci.yml` runs automatically on main pushes, not release-branch pushes. Dispatch it explicitly with `gh workflow run ci.yml --ref "release/<target>"`. Locate that dispatch run, verify its `head_sha` equals the release branch's remote HEAD, and require the full expected CI job set to pass, including `Quick Checks`, `Test and Lint`, and `End-to-End Tests (full merge gate)`. Treat a missing or skipped required lane as incomplete. A successful PR check or CI run for a different SHA does not qualify. If the branch advances or a run is cancelled, validate the new SHA again. Do not tag a candidate with failing or incomplete CI.
- Prefer protecting `release/*` against force pushes and deletion and requiring reviewed backport PRs. Check live rulesets rather than assuming the main ruleset covers release branches; report a missing release-branch rule as a gap.
- Set `PREVIEW_HASH` to the CI-validated release-branch commit and recheck the remote branch HEAD immediately before tagging. Report the branch, commit, and CI run URL; do not derive `PREVIEW_HASH` from a later `origin/main` fetch. When adopting an existing preview tag at this SHA, skip tag creation and continue Phase 3.
```bash
git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
git push origin "<preview-tag>"
```
Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Docker may publish exact preview image tags. Latest-channel, R2, and Helm publication must be skipped; installation references and the website announcement stay unchanged.
On a restart after a backport, re-run release-branch CI before setting the new `PREVIEW_HASH`. Use the next unused preview iteration when the validated commit differs from an existing preview tag; never move an existing tag.
@@ -1,118 +1,24 @@
---
name: rustfs-release-version-bump
description: "Prepare the version-file and release-asset bump for an exact RustFS alpha/beta/stable target, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
description: Prepare an exact RustFS Cargo version or align installation references after publication. Use for requested version bumps or the release workflow.
---
# RustFS Release Version Bump
# Release Version Files
Use this skill to prepare and verify release version files. Commit, push, and PR steps apply only when included in the user's delivery scope; publishing release tags belongs to `rustfs-release-publish`.
Require an exact target; ask if missing or ambiguous. Reject targets containing `-preview`: those are tag-only. Infer `post-release` only from an explicit request or parent workflow; otherwise use `prepare`. Delivery defaults to local edit/verify; commit, push, PR, and merge follow existing authorization and [Git rules](../../references/pull-requests.md).
Validated baseline: release pattern used in PR `#2957`.
| Stage | Files and constraints |
|---|---|
| `prepare` | Only `Cargo.toml` and `Cargo.lock`: workspace package, internal dependency, and workspace member versions become `<target>`. External dependencies stay unchanged. |
| `post-release` | Only `README.md`, `README_ZH.md`, `flake.nix`, `helm/rustfs/Chart.yaml`, and `rustfs.spec`. Cargo may already target the next release; leave it untouched. |
## Required inputs
Before installation edits, verify the exact non-preview Release, downloadable assets/source archive, pullable `rustfs/rustfs:<target>` image manifest, and published Helm chart. Block on missing artifacts. Do not downgrade defaults if a newer deliverable superseded the target. Preview preparation retains the previous published installation references; follow-up commits never replace either validated tag.
- Exact target version, for example `1.0.0-beta.4`.
- Delivery scope: local (`edit/verify`), git (`commit/push`), or GitHub
(`commit/push/PR`). Derive it from the conversation; when unspecified, prepare
and verify locally without blocking on a delivery question.
Packaging policy:
If target version is missing or ambiguous, stop and ask before editing.
- Docker image tags have no `v` prefix. Derive chart/app versions with `scripts/helm_chart_version.sh <target>` (`beta.N -> 0.N.0`; other versions unchanged). Helm CI derives versions from the final tag, so no early chart bump is required.
- RPM `Version` is numeric; `Release` is the prerelease suffix or `1` for stable. Expanded `Source0` and the unpack directory must resolve to the exact target including its suffix. Add the changelog entry using current git identity, date, and target; never copy an example identity/date.
- Changing these mappings requires explicit confirmation.
Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
Verification follows root tiers: `prepare` validates Cargo metadata with the updated lockfile and checks every workspace/internal version with no unrelated lockfile updates. `post-release` renders Helm with the verified image, runs `scripts/test_helm_chart_version.sh` for chart version edits, and checks README, Nix, and RPM references. Run `git diff --check`; report unresolved required checks as `BLOCKED` without unrelated fixes.
## Read before editing
- `AGENTS.md` (root and nearest path-specific files).
- `.github/pull_request_template.md` only when preparing a PR.
- Current branch status and diff against `origin/main`.
## Default release file scope
Treat the following file list as the default checklist for each release bump:
- `Cargo.toml`
- `Cargo.lock`
- `README.md`
- `README_ZH.md`
- `flake.nix`
- `helm/rustfs/Chart.yaml`
- `rustfs.spec`
Only drop a file when the current repository release process clearly no longer requires it.
## Hard release policy
- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
- Helm chart version mapping follows `beta.N -> 0.N.0`.
- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
- Do not change these rules without explicit confirmation.
## Step-by-step workflow
1. Confirm intent and isolate scope
- Use the exact target and delivery scope already supplied; ask only for a missing or ambiguous target or a material release-policy choice.
- Inspect current branch and ensure only release-related files are touched for this task.
2. Update workspace versions
- Bump `[workspace.package].version` in `Cargo.toml`.
- Bump internal workspace crate dependency versions in `Cargo.toml`.
- Update `Cargo.lock` so workspace package versions match target version.
- Re-scan for partial leftovers.
3. Update release assets
- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
- `flake.nix`: update package version to target version.
- `helm/rustfs/Chart.yaml`:
- `appVersion` = target version.
- `version` follows chart mapping rule, for example:
- `1.0.0-beta.3` -> `0.3.0`
- `1.0.0-beta.4` -> `0.4.0`
- `rustfs.spec`:
- Set `Release` to prerelease suffix (example `beta.4`).
- Add/update top changelog entry with exact format:
- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
- `- Update RPM package to RustFS 1.0.0-beta.4`
- Changelog identity and time must come from current environment:
- `git config --get user.name`
- `git config --get user.email`
- `date '+%a %b %d %Y'`
- Changelog version text must match target release version exactly.
4. Verify before shipping
- Run:
- `make pre-commit`
- If `make pre-commit` fails, fix task-attributable failures and rerun affected checks. Report unresolved required checks as `BLOCKED`; do not silently widen scope to fix unrelated issues.
5. Commit strategy (only when committing is authorized)
- Preferred split when both parts changed:
- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
- `chore(release): align release assets for <version>` for docs and packaging files.
- If user asks for one commit, use one commit.
- Stage only intended release files; do not include unrelated working tree changes.
6. Push and PR (only for the authorized delivery scope)
- Push branch:
- Use the user-requested or configured push remote: `git push -u <push-remote> <branch>` (first push), or `git push` when tracking is already configured.
- Create PR with template headings unchanged:
- `gh pr create --base main --head <branch> --title ... --body-file ...`
- PR title/body must be English.
- Use `N/A` for non-applicable template sections.
- Include verification commands and any `BLOCKED` reason clearly.
## Recommended check commands
- `git status --short --branch`
- `git diff --name-only origin/main...HEAD`
- `git diff --stat origin/main...HEAD`
- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
- `make pre-commit`
## Output contract
When using this skill, always report:
- Target version.
- Files changed.
- Any assumptions or uncertainties requiring confirmation.
- Verification result (`PASSED` or `BLOCKED`) with key evidence.
- Commit message(s) used.
- Push status and PR URL when GitHub flow is requested.
For authorized commits, keep stages separate: `chore(release): prepare <target>` or `chore(release): align installation references for <target>`. Report target/stage, changed files, verification, material uncertainty, and actual commit/push/PR state.
@@ -1,4 +0,0 @@
interface:
display_name: "RustFS Release Bump"
short_description: "Prepare RustFS release branches like PR #2957."
default_prompt: "Use $rustfs-release-version-bump to prepare and verify an exact RustFS release-version bump."
@@ -36,6 +36,21 @@ changed lesson.
## Finding Standard
Each finding includes severity, `file:line`, attacker prerequisites, concrete
input/path, impact, smallest safe fix, and a regression check. Do not exaggerate
unauthenticated impact when the actual issue requires authenticated low privilege.
Every candidate names the lower-trust principal, the boundary it crosses, the
affected resource, and the observable result. A best-practice gap or an impact
confined to the caller's own principal is hardening, not a candidate.
Report each candidate under exactly one verdict:
- `confirmed`: a complete `file:line` source trace and a bounded observed
result. Only confirmed findings carry severity; each also states attacker
prerequisites, concrete input/path, impact, smallest safe fix, and a
regression check.
- `needs_validation`: the trace is blocked on one exact unresolved fact, such
as deployment or infrastructure behavior. State that fact; assign no severity
and do not substitute an assumption.
- `rejected`: a disproved candidate, with the caller, invariant, or test that
disproves it.
Do not exaggerate unauthenticated impact when the actual issue requires
authenticated low privilege.
@@ -1,4 +0,0 @@
interface:
display_name: "Security Advisory Lessons"
short_description: "Apply advisory lessons in reviews."
default_prompt: "Use $security-advisory-lessons for a dedicated RustFS security review grounded in past advisories."
@@ -45,11 +45,14 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact.
### S3 object actions, copy, multipart, and upload policy validation
- `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID.
- `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
@@ -60,12 +63,14 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
- `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
- `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
- `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
- `GHSA-rhg6-jrf2-29c4`: WebDAV, FTP/FTPS, and SFTP used identity-policy checks with empty conditions and bypassed S3 bucket-policy explicit denies and condition-key denies. Lesson: alternate listeners must evaluate the same effective policy and trustworthy request context before storage access.
### Filesystem paths and object key traversal
- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
- `GHSA-g7fq-6xjm-mxf4`: versioned `DeleteObjects` accepted dot-segment keys, authorized the literal prefix, then wrote delete markers under a resolved sibling prefix. Lesson: validate each batch key before authorization and storage normalization, using the same object-name rules as single delete.
### Secrets, defaults, and cryptographic misuse
@@ -106,6 +111,8 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
### Object Lock and retention invariants
- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
- `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound.
### Serde deserialization and input validation
@@ -135,16 +142,17 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
## Minimum Regression Test Expectations
- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases; test bucket-policy explicit denies and condition-key denies on each enabled listener against an S3 control, and assert denied paths do not reach storage.
- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
- Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases.
- Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode.
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention.
@@ -1,57 +0,0 @@
---
name: test-coverage-improver
description: Analyze a supplied coverage report or perform an explicitly requested RustFS coverage assessment, rank uncovered risks, and propose focused tests. Do not trigger for ordinary implementation verification, a single regression test, documentation wording, or release preparation without a coverage request.
---
# Test Coverage Improver
Use this skill when you need a prioritized, risk-aware plan to improve tests from coverage results.
## Usage assumptions
- Focus scope is either changed lines/files, a module, or the whole repository.
- Reuse a supplied coverage artifact when its revision, scope, and format match.
- If required context is missing, call out assumptions explicitly before proposing work.
## Workflow
1. Define scope and baseline
- Derive the revision and scope from the request, diff, or supplied report.
- Default to the affected files/module; whole-workspace coverage requires that
scope in the request. Ask only if a wrong scope would change the result.
2. Obtain coverage evidence
- First inspect a matching existing artifact; do not regenerate it merely
because this skill was selected.
- If measurement is needed, read the Coverage section of
[the testing guide](../../../docs/testing/README.md#coverage), check disk
space/tool availability, and select package/test-scoped `cargo llvm-cov`
using the repository's nextest configuration. `make coverage` measures the
whole workspace (excluding E2E) and is only for that requested scope.
- Collect only metrics the report supports. Missing branch/changed-line
coverage is unknown, not zero.
- If measurement cannot run, continue with code-based test proposals and
mark measured coverage unverified; do not invent a coverage percentage.
3. Rank highest-risk gaps
- Prioritize changed code, branch coverage gaps, and low-confidence boundaries.
- Apply the risk rubric in [coverage-prioritization.md](references/coverage-prioritization.md).
- Report up to 5–8 evidenced gaps; do not pad a small scope.
- For each gap, capture: file, lines, uncovered branches, and estimated risk score.
4. Propose high-impact tests
- For each gap, name the behavior and regression, distinguishing assertions,
relevant normal/edge/failure cases, necessary setup, and estimated effort.
- Include only scenarios and setup that apply; reuse shared fixture details.
5. Close with validation plan
- State which gaps remain after proposals.
- Give a scoped verification command and behavior-based acceptance criterion;
use a coverage threshold only when the task or repository requires one.
- List assumptions or blockers (environment, fixtures, flaky dependencies).
## Report
Summarize the supported metrics, then combine each ranked gap with its proposed
test and validation. Include source lines only when supplied or inspected;
mark missing metrics or locations as unknown. Do not duplicate gaps and tests
in separate templates or fill empty categories for an otherwise small report.
@@ -1,4 +0,0 @@
interface:
display_name: "Test Coverage Improver"
short_description: "Find top uncovered risk areas and propose high-impact tests."
default_prompt: "Use $test-coverage-improver to analyze coverage for the requested scope, reuse matching reports, and propose tests for evidenced risks."
@@ -1,25 +0,0 @@
# Coverage Gap Prioritization Guide
Use this rubric for each uncovered area.
Score = (Criticality × 2) + CoverageDebt + (Volatility × 0.5)
- Criticality:
- 5: authz/authn, data-loss, payment/consistency path
- 4: state mutation, cache invalidation, scheduling
- 3: error handling + fallbacks in user-visible flows
- 2: parsing/format conversion paths
- 1: logging-only or low-impact utilities
- CoverageDebt:
- 0: 0–5 uncovered lines
- 1: 6–20 uncovered lines
- 2: 21–40 uncovered lines
- 3: 41+ uncovered lines
- Volatility:
- 1: stable legacy code with few recent edits
- 2: changed in last 2 releases
- 3: touched in last 30 days or currently in active PR
Sort by score descending, then by business impact.
-5
View File
@@ -1,5 +0,0 @@
# Bound individual tool outputs retained in context; retrieve relevant ranges
# from task-owned log files when more evidence is needed.
# https://learn.chatgpt.com/docs/config-file/config-reference
tool_output_token_limit = 4000
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-linux=21ab66fd172018682ce825a53f26aa8f286d57f3f687d8616d1850e56474dcd8
sha256-darwin=21ab66fd172018682ce825a53f26aa8f286d57f3f687d8616d1850e56474dcd8
sha256-linux=cf31ed27e4e6bf42da5bf81f248d9d2619ac8e8615186cdacb30bd63996d7607
sha256-darwin=cf31ed27e4e6bf42da5bf81f248d9d2619ac8e8615186cdacb30bd63996d7607
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=9e9687ded961aa4e619a15def6e1ee0a5153be05423d009ee7a58126624c83e7
sha256-linux=364666693c2bf258997643c9d87e07e7b7501e58175f190d201d53c81946eb20
sha256-darwin=7893834728e2dadb98110c2aa018e6aa150c882c8a39c753e8095c3aef94cf0e
sha256-linux=fb95ac8ba4b68aec9182fe32a59923e6b0d37b45a62ce87ea7db0dc3c40375e6
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=83a7dcaffd5a789517ae9f02a224f66a9713937885cff96fca2ad7e216f197ae
sha256-linux=626c10f8c964507ff987b6c86069e9019dc6d2ae7fb02db9be5df5aa8cc5145b
sha256-darwin=347a0d4e5e547d5388703062ce66a66167a845b6e57ec591ccb4c732a567c68d
sha256-linux=4f9efa47194279185bb68249e82c788fa147abe944b6c764050e3d47f31e7926
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=55534a97fbd376f64c8f6c341d319017d11ff77cad6da8629a1a7f6a874e0315
sha256-linux=c06fb8c19aed6f388b9dc61cb8251b7a44f8561a9bf764ad2b9e635598f8dc17
sha256-darwin=a2cb67d91eb0d093928371d6e84a84b2dd41b2ba975a28094e81879bd65838ad
sha256-linux=82db0561cf9b111f11782d647d0859c9d2a44c5c68481b5e39c48505828bd392
+1 -1
View File
@@ -1 +1 @@
sha256=e4ee3898b840a8374cb02c64342cc15f8d7ecd1becaf4b4a2a4657290f9e7ff1
sha256=4eac269f9a59b2f9be4e7a87ea6aa38454f5ebe10ede35c9b8d4218137d2cd35
+1 -1
View File
@@ -1 +1 @@
sha256=548a58d74f3cb3b5a1ab9dcd4d3f8625b9ddebe847a2e14de1d08c3b2e6292eb
sha256=aa603dbbae2e0dc0803489a88887a630c542a6949466c2b0e73b8a6b5e57fcaa
+305
View File
@@ -55,6 +55,281 @@
"invariant": "odm-source-contract-gcs",
"suite": "rustfs",
"name": "on_demand_migration::gcs::tests::gcs_native_backend_satisfies_the_shared_backend_contract"
},
{
"invariant": "rollback-inspection",
"suite": "rustfs-ecstore",
"name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_preserves_overwrite_data_dirs_and_staging"
},
{
"invariant": "rollback-closed-inspection-channel",
"suite": "rustfs-ecstore",
"name": "set_disk::core::io_primitives::tests::rename_rollback_closed_heal_receiver_preserves_recovery_material"
},
{
"invariant": "rollback-cancellation",
"suite": "rustfs-ecstore",
"name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_cancelled_task_is_not_success"
},
{
"invariant": "rollback-completion-equivalence",
"suite": "rustfs-ecstore",
"name": "set_disk::core::io_primitives::tests::rename_rollback_incomplete_matches_early_ack_and_full_wait_after_reopen"
},
{
"invariant": "rollback-physical-owner",
"suite": "rustfs-ecstore",
"name": "set_disk::core::io_primitives::tests::rename_rollback_children_keep_namespace_ownership_after_coordinator_panic"
},
{
"invariant": "late-ack-retains-commit",
"suite": "rustfs-ecstore",
"name": "set_disk::core::io_primitives::tests::rename_data_early_ack_post_mutation_tail_error_never_rolls_back_commit"
},
{
"invariant": "tail-drained-cas",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::put_object_tmp_cleanup_tests::tail_drained_put_waits_for_tail_and_allows_immediate_cas"
},
{
"invariant": "seeded-put-tail-pressure",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::put_object_tmp_cleanup_tests::seeded_put_completion_pressure_drains_every_real_tail"
},
{
"invariant": "borrowed-lock-tail",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::put_object_tmp_cleanup_tests::no_lock_put_waits_for_rename_tail_under_outer_guard"
},
{
"invariant": "cancelled-writer-owner",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::put_object_tmp_cleanup_tests::tail_drained_put_owned_commit_survives_waiter_cancellation"
},
{
"invariant": "get-body-reconstruction",
"suite": "rustfs-ecstore",
"name": "ecstore_validation_blackbox::blackbox_get_restores_body_after_one_shard_file_is_removed"
},
{
"invariant": "range-reconstruction",
"suite": "rustfs-ecstore",
"name": "ecstore_validation_blackbox::blackbox_range_read_restores_exact_slice_with_one_offline_disk"
},
{
"invariant": "multipart-abort-linearization",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::multipart::tests::abort_and_complete_linearize_for_plain_sse_and_legacy_layouts"
},
{
"invariant": "multipart-write-quorum",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::multipart::tests::abort_enforces_delete_write_quorum_boundary"
},
{
"invariant": "multipart-physical-owner",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::multipart::tests::complete_multipart_timeout_keeps_namespace_owner_until_physical_publication"
},
{
"invariant": "restore-abort-cleanup",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::transition_commit_failure_tests::multipart_restore_aborts_every_post_create_failure"
},
{
"invariant": "legacy-metadata",
"suite": "rustfs-filemeta",
"name": "filemeta::test::test_issue_2288_legacy_xlmeta_compatibility"
},
{
"invariant": "legacy-nil-version",
"suite": "rustfs-filemeta",
"name": "filemeta::test::test_into_fileinfo_reads_legacy_nil_uuid_inline_key"
},
{
"invariant": "commit-phase-schedule",
"suite": "rustfs-ecstore",
"name": "set_disk::core::io_primitives::tests::commit_fault_schedule_keeps_multiple_disk_phases_independent"
},
{
"invariant": "commit-phase-ownership",
"suite": "rustfs-ecstore",
"name": "set_disk::core::io_primitives::tests::commit_fault_schedule_rejects_duplicate_checkpoint_without_disarming_owner"
},
{
"invariant": "metadata-observation-all-four-slot-states-and-arrival-orders-preserve-reduction",
"suite": "rustfs-ecstore",
"name": "set_disk::core::metadata_quorum::tests::metadata_observation_all_four_slot_states_and_arrival_orders_preserve_reduction"
},
{
"invariant": "metadata-observation-mixed-current-identity-arrivals-preserve-selection",
"suite": "rustfs-ecstore",
"name": "set_disk::core::metadata_quorum::tests::metadata_observation_mixed_current_identity_arrivals_preserve_selection"
},
{
"invariant": "metadata-observation-pending-newer-version-and-same-time-different-directory-force-full-wait",
"suite": "rustfs-ecstore",
"name": "set_disk::core::metadata_quorum::tests::metadata_observation_pending_newer_version_and_same_time_different_directory_force_full_wait"
},
{
"invariant": "metadata-observation-duplicate-shard-cannot-supply-a-read-reserve",
"suite": "rustfs-ecstore",
"name": "set_disk::core::metadata_quorum::tests::metadata_observation_duplicate_shard_cannot_supply_a_read_reserve"
},
{
"invariant": "metadata-observation-null-versions-markers-and-invalid-success-keep-their-meaning",
"suite": "rustfs-ecstore",
"name": "set_disk::core::metadata_quorum::tests::metadata_observation_null_versions_markers_and_invalid_success_keep_their_meaning"
},
{
"invariant": "metadata-cache-real-get-and-range-preserve-body-across-read-modes",
"suite": "rustfs-ecstore",
"name": "set_disk::read::metadata_cache_tests::metadata_cache_real_get_and_range_preserve_body_across_read_modes"
},
{
"invariant": "late-metadata-refresh-rejects-changed-generation-and-mapped-shards",
"suite": "rustfs-ecstore",
"name": "set_disk::read::metadata_cache_tests::late_metadata_refresh_rejects_changed_generation_and_mapped_shards"
},
{
"invariant": "late-full-get-rejects-changed-object-generation",
"suite": "rustfs-ecstore",
"name": "set_disk::prepared_get_object_metadata_tests::non_inline_two_phase_get_rejects_generation_change_before_late_refresh"
},
{
"invariant": "list-objects-shared-corpus-static-namespace",
"suite": "rustfs-ecstore",
"name": "store::list_objects::differential_tests::list_objects_shared_corpus_static_namespace"
},
{
"invariant": "list-objects-shared-corpus-cross-subdirectory-limit",
"suite": "rustfs-ecstore",
"name": "store::list_objects::differential_tests::list_objects_shared_corpus_cross_subdirectory_limit"
},
{
"invariant": "list-objects-shared-corpus-max-and-max-plus-one",
"suite": "rustfs-ecstore",
"name": "store::list_objects::differential_tests::list_objects_shared_corpus_max_and_max_plus_one"
},
{
"invariant": "list-objects-shared-corpus-fixed-seed-marker-and-prefix",
"suite": "rustfs-ecstore",
"name": "store::list_objects::differential_tests::list_objects_shared_corpus_fixed_seed_marker_and_prefix"
},
{
"invariant": "list-objects-shared-corpus-offline-corrupt-and-minority-stale",
"suite": "rustfs-ecstore",
"name": "store::list_objects::differential_tests::list_objects_shared_corpus_offline_corrupt_and_minority_stale"
},
{
"invariant": "list-objects-shared-corpus-versions-delete-markers-and-free-versions",
"suite": "rustfs-ecstore",
"name": "store::list_objects::differential_tests::list_objects_shared_corpus_versions_delete_markers_and_free_versions"
},
{
"invariant": "list-objects-shared-corpus-orphan-cleanup-requires-authoritative-evidence",
"suite": "rustfs-ecstore",
"name": "store::list_objects::differential_tests::list_objects_shared_corpus_orphan_cleanup_requires_authoritative_evidence"
},
{
"invariant": "list-objects-shared-corpus-consumer-drop-joins-real-disk-producer",
"suite": "rustfs-ecstore",
"name": "store::list_objects::differential_tests::list_objects_shared_corpus_consumer_drop_joins_real_disk_producer"
},
{
"invariant": "list-objects-shared-corpus-cancelled-quorum-walk-releases-senders-and-disk-lock",
"suite": "rustfs-ecstore",
"name": "store::list_objects::differential_tests::list_objects_shared_corpus_cancelled_quorum_walk_releases_senders_and_disk_lock"
},
{
"invariant": "list-objects-shared-corpus-enabled-index-contracts",
"suite": "rustfs-ecstore",
"name": "store::list_objects::differential_tests::list_objects_shared_corpus_enabled_index_contracts"
},
{
"invariant": "borrowed-context-boundary",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::put_object_tmp_cleanup_tests::borrowed_write_context_rejects_wrong_namespace_and_cache_flag"
},
{
"invariant": "borrowed-put-cancellation",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::put_object_tmp_cleanup_tests::borrowed_tail_drained_put_keeps_owner_after_caller_cancel"
},
{
"invariant": "borrowed-mpu-cancellation",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::multipart::tests::borrowed_complete_keeps_namespace_owner_after_caller_cancel"
},
{
"invariant": "pool-cas-owner",
"suite": "rustfs-ecstore",
"name": "core::pools::pools_tests::pool_meta_cas_context_retains_actual_owner_for_exact_target"
},
{
"invariant": "pool-cas-released-owner",
"suite": "rustfs-ecstore",
"name": "core::pools::pools_tests::pool_meta_cas_rejects_released_owner"
},
{
"invariant": "copy-tail-drained",
"suite": "rustfs-ecstore",
"name": "store::object::tests::borrowed_copy_tail_drained_waits_for_rename_tail"
},
{
"invariant": "catalog-borrowed-publication",
"suite": "rustfs",
"name": "admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition"
},
{
"invariant": "directory-marker-quorum",
"suite": "rustfs-filemeta",
"name": "metacache::tests::resolve_directory_marker_and_prefix_preserve_quorum_in_both_orders"
},
{
"invariant": "logical-prefix-limit",
"suite": "rustfs-ecstore",
"name": "store::list_objects::test::list_path_gather_results_counts_common_prefixes_before_page_limit"
},
{
"invariant": "quota-same-name-owner-and-disk-authority",
"suite": "rustfs-ecstore",
"name": "bucket::metadata_sys::tests::quota_snapshot_reuses_same_name_owner_and_rereads_disk_authority"
},
{
"invariant": "quota-snapshot-scope-and-owner",
"suite": "rustfs-ecstore",
"name": "bucket::metadata_sys::tests::quota_snapshot_rejects_wrong_scope_incarnation_and_transaction_guard"
},
{
"invariant": "quota-snapshot-lifecycle-and-incarnation",
"suite": "rustfs-ecstore",
"name": "bucket::metadata_sys::tests::quota_snapshot_rejects_lifecycle_loss_during_read_and_changed_disk_incarnation"
},
{
"invariant": "marker-purge-receipt-reopen",
"suite": "rustfs-ecstore",
"name": "set_disk::ops::object::marker_purge_receipt_tests::quorum_marker_purge_receipt_survives_reopen_with_stale_member"
},
{
"invariant": "external-minio-shards",
"suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
"name": "pinned_minio_shards_decode_and_reconstruct_plaintext"
},
{
"invariant": "external-legacy-shards",
"suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
"name": "pinned_legacy_shards_decode_and_reconstruct_plaintext"
},
{
"invariant": "external-shard-bitrot",
"suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
"name": "pinned_erasure_shards_reject_bitrot_corruption"
},
{
"invariant": "external-shard-quorum-codec",
"suite": "rustfs-ecstore::pinned_erasure_fixtures_test",
"name": "pinned_erasure_shards_reject_insufficient_quorum_and_wrong_codec"
}
],
"fixtures": [
@@ -82,6 +357,36 @@
"path": "crates/ecstore/tests/fixtures/minio/bucket_metadata_full.xlmeta.hex",
"sha256": "3b6de589519c08a1614c8bd409bb8199c17d42043861b07bce513075e6fbfc12",
"source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/minio/README.md"
},
{
"path": "crates/filemeta/tests/fixtures/issue_2288_legacy_xlmeta.hex",
"sha256": "b3e8d32f9950a38b5dc71f679519cc830a2b2a807e8b0b39f78840b238d8eacf",
"source": "Legacy xl.meta captured for rustfs/backlog#2288; filemeta::test::test_issue_2288_legacy_xlmeta_compatibility"
},
{
"path": "crates/filemeta/tests/fixtures/issue_2265_legacy_meta_v2_object.hex",
"sha256": "89dd7ec92d16e54b5d31e0c49c9edffff8a7aa58890aaeff59bc2ff84da0a62f",
"source": "Legacy metadata v2 object captured for rustfs/backlog#2265; crates/filemeta/src/test_data.rs"
},
{
"path": "crates/ecstore/tests/fixtures/list_namespace_keys.json",
"sha256": "5afddb60565deba0938e617470255563f0e15a15fab699645c59bb7ccc84898e",
"source": "Shared static LIST namespace corpus; ECStore differential tests and ODM list_through_static_namespace_boundary_matrix"
},
{
"path": "crates/ecstore/tests/fixtures/erasure-shards/minio/manifest.json",
"sha256": "c398f8dcb76903f6cf8eee07931c76592895ced201ff9d59ed4ff9b0ead2b13c",
"source": "MinIO RELEASE.2025-07-23T15-54-02Z; crates/ecstore/tests/fixtures/erasure-shards/README.md"
},
{
"path": "crates/ecstore/tests/fixtures/erasure-shards/legacy/manifest.json",
"sha256": "dce397d5b2f42748eb5091b4355bb9d133a50cbba003626fafac28095dbe540b",
"source": "RustFS 1.0.0-alpha.37; crates/ecstore/tests/fixtures/erasure-shards/README.md"
},
{
"path": "crates/ecstore/tests/fixtures/erasure-shards/plaintext.bin",
"sha256": "b5a83332327961a31b65a8731c64559fd04ff8ac9f6124022201dbbdba5ffa01",
"source": "Independent fixed SHA256 counter-stream oracle; crates/ecstore/tests/fixtures/erasure-shards/README.md"
}
]
}
+5
View File
@@ -70,6 +70,11 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
@echo "📦 Checking s3s footprint ratchet..."
./scripts/check_s3s_footprint.sh
.PHONY: test-count-ratchet-check
test-count-ratchet-check: ## Check no workspace crate lost tests or gained #[ignore] against the committed baseline
@echo "🧪 Checking per-crate test-count ratchet..."
./scripts/check_test_count_ratchet.sh
.PHONY: fips-wording-check
fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
@echo "📣 Checking cryptographic capability wording guard..."
+2 -2
View File
@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY: pre-commit
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY: pre-pr
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY: dev-check
+8
View File
@@ -24,6 +24,7 @@ TEST_THREADS ?= 1
.PHONY: script-tests
script-tests: ## Run shell script tests
@echo "Running script tests..."
trap 'status=$$?; printf "ERROR: script-tests failed (exit %s): %s\n" "$$status" "$$BASH_COMMAND" >&2; exit "$$status"' ERR
./scripts/test_build_rustfs_options.sh
./scripts/test_docker_runtime_timezone.sh
./scripts/test_entrypoint_credentials.sh
@@ -49,8 +50,14 @@ script-tests: ## Run shell script tests
./scripts/test_fuzz_runner.sh
./scripts/test_python_bin.sh
./scripts/check_embedded_secrets.sh --self-test
./scripts/check_s3s_footprint.sh --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_scheduled_validation_freshness.py --self-test
@@ -60,6 +67,7 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_functional_chain_health.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_runner_tools.py
bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
$(RUSTFS_PYTHON_BIN) ./scripts/check_object_data_cache_follower_samples.py --self-test
./scripts/validate_object_data_cache_cold_stampede.sh --self-test
+237 -15
View File
@@ -64,14 +64,72 @@ command = ['sh', '-c', 'echo RUST_MIN_STACK=33554432 >> "$NEXTEST_ENV"']
command = ['sh', '-c', 'echo RUST_MIN_STACK=33554432 >> "$NEXTEST_ENV"']
# --- default profile (local): serialize the flaky groups, never retry --------
# These real-disk progress oracles exercise fsync, quorum IO, and crash/replay
# boundaries. Reserve the run's capacity so unrelated fixtures cannot starve
# their existing deadlines and typed-failure assertions (rustfs/backlog#2706).
# Keep each test's internal concurrency and zero-retry policy unchanged.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(/^connect::diagnostics::schedule::tests::/)'
threads-required = "num-test-threads"
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & (test(/^set_disk::tests::streaming_get_blocks_concurrent_/) | test(=store::object::tests::retired_marker_converges_after_bucket_recreation_on_sixteen_disks))'
threads-required = "num-test-threads"
# These real-disk lock probes must finish durable IO within production's 5s
# acquisition budget. Reserve capacity from sibling test processes while
# preserving the probes' internal concurrency, deadlines, and assertions.
[[profile.default.overrides]]
filter = '(package(rustfs-ecstore) & test(=core::pools_test::decommission_lock_order_tests::active_decommission_migration_does_not_invert_capacity_and_delete_object_tag_locks)) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
threads-required = "num-test-threads"
# These eight-disk decommission fixtures exercise concurrent retries,
# free-version cleanup, and real capacity probes under production lock deadlines.
# Reserve the run's capacity so unrelated IO cannot exhaust those budgets.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & (test(/^store::init::tests::decommission_entry_(retries_source_changed_without_canceling_other_bucket|skips_cleanup_only_marker_when_free_version_is_present)$/) | test(=core::pools::tests::decommission_capacity_mutation_limits_prewrite_sampling_to_target))'
threads-required = "num-test-threads"
# This registration/backup lock probe uses an isolated eight-disk store.
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.default.overrides]]
filter = 'package(rustfs-heal) & (binary(mrf_partial_write_test) | (binary(mrf_pipeline_test) & test(/^journal_replay_survives_/)) | test(=admin_selector_survives_sigkill_with_original_scope_and_receipts))'
threads-required = "num-test-threads"
# Native scanner backlog fixtures include 24-set and pool-expansion cases.
# Reserve capacity across test processes while preserving each scenario's fixed
# budget, assertions, and internal concurrency.
[[profile.default.overrides]]
filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
threads-required = "num-test-threads"
[[profile.default.scripts]]
filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
setup = 'ecstore-large-stack'
# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
# The admin target-repair fixtures compose the full bucket delete and recreate
# path on a real store; measured with RUST_MIN_STACK on 2026-10-15 they need
# 2016-2040 KiB of the 2 MiB libtest default and overflowed on Linux CI.
[[profile.default.scripts]]
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
setup = 'ecstore-base-stack'
[[profile.default.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
# These handler/store fixtures and the relocated-pool GET compose deep storage
# futures in libtest threads; on Linux they overflow the default 2 MiB stack.
[[profile.default.scripts]]
filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
setup = 'ecstore-base-stack'
[[profile.default.scripts]]
filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
setup = 'lifecycle-large-stack'
@@ -82,6 +140,13 @@ setup = 'lifecycle-large-stack'
filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
setup = 'ecstore-large-stack'
# These real-disk scanner fixtures assert quiet-workload publication and
# durable MRF admission. Reserve the pool while their bounded CAS/fsync
# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
[[profile.default.overrides]]
filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
threads-required = "num-test-threads"
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
test-group = 'ecstore-serial-flaky'
@@ -107,6 +172,16 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
test-group = 'ecstore-serial-flaky'
# Serialize the replacement bucket metadata heal test. It builds a real-disk
# hermetic erasure set, writes bucket metadata, then asserts byte-identical
# xl.meta readback after a rejected delayed repair. Under parallel load,
# concurrent ecstore tests mutate the process-wide disk registry and alter
# the serialized timestamp/disk-identity fields, flipping the assertion.
# Preventive serialization only, no retries.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
test-group = 'ecstore-serial-flaky'
# Serialize the metadata-cache generation-retirement pair. Both carry
# #[serial(metadata_cache_invalidation_probe)] — a no-op across nextest's
# process boundary — and assert get_object_metadata_cache generation
@@ -132,10 +207,38 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
test-group = 'embedded-test-ports'
# The real object probe includes bucket creation and cleanup in its short
# measurement budget. Keep competing storage fixtures outside that budget.
# Registration, inventory delivery, and real drive/object probes include
# filesystem or HTTP IO in short deadlines. Reserve capacity so sibling test
# processes cannot exhaust those budgets. Keep deadlines and assertions intact.
# The short HTTP fixtures are tracked in rustfs/backlog#2714.
[[profile.default.overrides]]
filter = 'package(rustfs) & binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
threads-required = "num-test-threads"
# This cancellation probe hashes the serving executable before acquiring its
# collector lease. Keep competing test processes out of its 30s startup budget.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(=connect::diagnostics::trace_runtime::tests::local_runtime_profile_cancellation_waits_for_lease_release)'
threads-required = "num-test-threads"
# These real IAM/state fixtures keep production lock-acquisition limits while
# peers complete durable IO. Reserve capacity from unrelated test processes;
# the shared fixture's durable_iam_state_ prefix keeps the whole family together.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::durable_iam_state_/)'
threads-required = "num-test-threads"
# Real target-repair and rebalance-stop fixtures commit durable metadata while
# peers wait on production locks or bounded cancellation checks. Reserve IO
# capacity for each fixture; its internal races and deadlines remain intact.
[[profile.default.overrides]]
filter = 'package(rustfs) & test(/^admin::handlers::(replication::target_repair_tests::|rebalance::rebalance_handler_tests::real_admin_stop_)/)'
threads-required = "num-test-threads"
# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
# Their existing deadlines, assertions, and retry policy remain in force.
[[profile.default.overrides]]
filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
threads-required = "num-test-threads"
# Serialize the durable manual-transition checkpoint test across nextest's
@@ -218,12 +321,25 @@ test-group = 'e2e-cluster-nightly'
filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
test-group = 'e2e-vault'
# These server-heavy smoke cases perform 1k+ sequential S3 writes or run the
# scanner and a remote tier server together. Reserve the full test budget so
# unrelated E2E processes cannot starve their storage and polling deadlines.
[[profile.default.overrides]]
filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
threads-required = "num-test-threads"
# This four-disk, 65-member rollback probe already drives up to 32 concurrent
# durable deletions. Reserve this nextest run's capacity for its progress oracle.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & test(=store::init::tests::dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member)'
threads-required = "num-test-threads"
# These short-deadline real-disk progress probes need the test run's capacity
# for scanner and object-lock handoffs; keep their assertions and deadlines.
[[profile.default.overrides]]
filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
threads-required = "num-test-threads"
# ---------------------------------------------------------------------------
# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
# ---------------------------------------------------------------------------
@@ -239,14 +355,71 @@ fail-fast = false
# marker is the observable signal the flake policy is built around.
path = "junit.xml"
# Match the local durable-progress reservations. All members, deadlines,
# assertions, and internal concurrency remain enabled without retries.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(/^connect::diagnostics::schedule::tests::/)'
threads-required = "num-test-threads"
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & (test(/^set_disk::tests::streaming_get_blocks_concurrent_/) | test(=store::object::tests::retired_marker_converges_after_bucket_recreation_on_sixteen_disks))'
threads-required = "num-test-threads"
# These real-disk lock probes must finish durable IO within production's 5s
# acquisition budget. Reserve capacity from sibling test processes while
# preserving the probes' internal concurrency, deadlines, and assertions.
[[profile.ci.overrides]]
filter = '(package(rustfs-ecstore) & test(=core::pools_test::decommission_lock_order_tests::active_decommission_migration_does_not_invert_capacity_and_delete_object_tag_locks)) | (package(rustfs-heal) & test(=heal::mrf_queue::partial_write::tests::durable_retry_submission_error_keeps_checkpoint_and_uses_backoff))'
threads-required = "num-test-threads"
# These eight-disk decommission fixtures exercise concurrent retries,
# free-version cleanup, and real capacity probes under production lock deadlines.
# Reserve the run's capacity so unrelated IO cannot exhaust those budgets.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & (test(/^store::init::tests::decommission_entry_(retries_source_changed_without_canceling_other_bucket|skips_cleanup_only_marker_when_free_version_is_present)$/) | test(=core::pools::tests::decommission_capacity_mutation_limits_prewrite_sampling_to_target))'
threads-required = "num-test-threads"
# This registration/backup lock probe uses an isolated eight-disk store.
# Reserve capacity so sibling disk-heavy tests cannot consume its 2s
# registration budget; retain the queued backup and deadlock assertions.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits)'
threads-required = "num-test-threads"
[[profile.ci.overrides]]
filter = 'package(rustfs-heal) & (binary(mrf_partial_write_test) | (binary(mrf_pipeline_test) & test(/^journal_replay_survives_/)) | test(=admin_selector_survives_sigkill_with_original_scope_and_receipts))'
threads-required = "num-test-threads"
# Native scanner backlog fixtures include 24-set and pool-expansion cases.
# Reserve capacity across test processes while preserving each scenario's fixed
# budget, assertions, and internal concurrency.
[[profile.ci.overrides]]
filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
threads-required = "num-test-threads"
# Keep short-deadline real-disk progress probes from competing with unrelated
# tests for scanner and object-lock capacity. Assertions and deadlines stay on.
[[profile.ci.overrides]]
filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
threads-required = "num-test-threads"
[[profile.ci.scripts]]
filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
setup = 'ecstore-large-stack'
# These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
[[profile.ci.scripts]]
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
setup = 'lifecycle-large-stack'
@@ -257,6 +430,13 @@ setup = 'lifecycle-large-stack'
filter = 'package(rustfs-heal) & test(=replacement_pool_metadata_follows_real_two_set_placement)'
setup = 'ecstore-large-stack'
# These real-disk scanner fixtures assert quiet-workload publication and
# durable MRF admission. Reserve the pool while their bounded CAS/fsync
# handoffs run so unrelated disk fanouts cannot consume that fixture budget.
[[profile.ci.overrides]]
filter = 'package(rustfs-scanner) & (test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(=scanner_io::tests::multi_pool_scanner_cycle_publishes_combined_usage) | test(=scanner_io::tests::pending_put_commit_keeps_scanner_walk_live_without_authoritative_usage) | test(=scanner_folder::tests::mrf_ownership::checkpoint_fixture_metadata_heal_reaches_budgeted_tail))'
threads-required = "num-test-threads"
# ===========================================================================
# QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
#
@@ -282,13 +462,6 @@ retries = 2
filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
test-group = 'ecstore-serial-flaky'
# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
# on producer/consumer timing windows that stretch past the budget on loaded
# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(walk_dir_does_not_charge_consumer_backpressure_to_the_stall_budget)'
retries = 2
# Serialize the relocated-pool GET resume regression under the ci profile too
# (see the matching default-profile override near the top). No longer a
# quarantine: the fixture race (rustfs#6701/rustfs#6703) was fixed by #6707,
@@ -320,6 +493,11 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::heal::heal_result_report_tests::/)'
test-group = 'ecstore-serial-flaky'
# Serialize the replacement bucket metadata heal test under the ci profile too.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(replacement_bucket_metadata_respects_pool_ownership_and_required_records)'
test-group = 'ecstore-serial-flaky'
# Serialize the metadata-cache generation-retirement pair under the ci
# profile too (see the matching default-profile override near the top). Not a
# quarantine: no retries.
@@ -334,7 +512,30 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
test-group = 'embedded-test-ports'
[[profile.ci.overrides]]
filter = 'package(rustfs) & binary(connect_perf_object) & test(=real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put)'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
threads-required = "num-test-threads"
# Preserve the cancellation probe's startup budget and assertions in CI too.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(=connect::diagnostics::trace_runtime::tests::local_runtime_profile_cancellation_waits_for_lease_release)'
threads-required = "num-test-threads"
# Keep the same IAM/state fixture capacity reservation in CI without changing the
# production lock deadline, internal concurrency, assertions, or retry policy.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(/^admin::handlers::site_replication::tests::durable_iam_state_/)'
threads-required = "num-test-threads"
# Match the real admin fixture capacity reservation without retries or changes
# to production lock deadlines, cancellation barriers, or assertions.
[[profile.ci.overrides]]
filter = 'package(rustfs) & test(/^admin::handlers::(replication::target_repair_tests::|rebalance::rebalance_handler_tests::real_admin_stop_)/)'
threads-required = "num-test-threads"
# Reserve capacity for these short heartbeat TLS/fsync probes (rustfs#8129).
# Their existing deadlines, assertions, and retry policy remain in force.
[[profile.ci.overrides]]
filter = 'package(rustfs) & binary(connect_heartbeat) & test(/^(restart_replays_pending_request_then_advances_sequence|restart_replays_exact_legacy_capabilities_with_and_without_jobs)$/)'
threads-required = "num-test-threads"
# Serialize the durable manual-transition checkpoint test under the ci profile
@@ -461,13 +662,24 @@ test-group = 'ecstore-serial-flaky'
# the PR-lane e2e coverage for scanner usage snapshots consumed by quota and
# admin surfaces. It uses the same single-node, random-port, isolated-temp-dir
# fixture as the existing smoke modules.
#
# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
# starts two single-node servers from the same binary, one per
# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
# SDK script covering every operation of `impl S3 for FS` against both. Its
# comparator unit tests need no server. It meets the admission criteria
# unchanged; the module clause admits a new differential test without another
# filter edit.
[profile.e2e-smoke]
default-filter = """
package(e2e_test) & (
test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat|data_usage)_test::|^fake_s3_target::/)
| test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
| test(/^reliant::lifecycle::/)
| test(/^conditional_delete_objects_test::/)
| test(/^reliant::tiering::/)
| test(/^dual_stack_differential::/)
| test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
| test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
)
"""
@@ -476,13 +688,16 @@ fail-fast = false
[profile.e2e-smoke.junit]
path = "junit.xml"
# The pagination boundary cases can stall when a server/listing regression
# prevents the continuation request from completing. Keep the timeout scoped
# to those known failure modes so legitimate lifecycle/tiering waits retain
# their test-level timing budget.
# Bound the whole pagination fixture, including startup, seeding, and cleanup;
# its LIST deadline begins only after the seed objects have been written.
[[profile.e2e-smoke.overrides]]
filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
threads-required = "num-test-threads"
[[profile.e2e-smoke.overrides]]
filter = 'package(e2e_test) & test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry)'
threads-required = "num-test-threads"
# ---------------------------------------------------------------------------
# e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
@@ -692,6 +907,7 @@ default-filter = """
& !test(/^replication_extension_test::/)
& !test(/^replication_target_matrix_test::/)
& !test(/^on_demand_migration::(concurrency_test|fault_test|interop_test|real_source_test)::/)
& !test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
"""
fail-fast = false
@@ -720,3 +936,9 @@ test-group = 'e2e-cluster-nightly'
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & (test(/^kms::kms_vault_test::/) | test(/^kms::kms_rekey_sweep_test::/) | test(/^kms::configured_roundtrip_test::test_configured_vault_kms_admin_and_versioned_cleanup$/))'
test-group = 'e2e-vault'
# Keep the storage-heavy smoke cases isolated in the full suite too. They
# start multiple server fixtures and issue many sequential S3 operations.
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
threads-required = "num-test-threads"
+37
View File
@@ -0,0 +1,37 @@
# Files that may still reference s3s (`s3s::` or `s3s_sigv4`) when Phase 1 of
# the gateway migration ends (rustfs/backlog#2734 task T0.6, rustfs/backlog#2741).
# Consumed only by `scripts/check_s3s_footprint.sh --mode allowlist`.
#
# One entry per line: a repository-relative file path or a git `:(glob)`
# pathspec. `**` matches across directories, `*` stays within one path
# component, and a directory entry matches everything under it. Blank lines
# and lines starting with `#` are ignored; an entry may not contain whitespace,
# so a note belongs on its own comment line.
#
# Every Rust file listed by `git grep -l '\bs3s\(::\|_sigv4\)' -- '*.rs'` that
# no entry matches fails the allowlist mode. Task T1.9 switches CI to that
# mode; until then `--dry-run` prints the files still outside this list, which
# is the Phase 1 tracking list.
#
# Never add crates/** other than crates/e2e_test: the engine crates leave s3s
# entirely, and the script rejects such an entry. Only the s3s-facing edge of
# the rustfs binary and the e2e oracle keep it.
# rustfs binary: the s3s-facing edge that survives Phase 1
rustfs/src/storage/ecfs.rs
rustfs/src/storage/s3_compat/**
rustfs/src/storage/access.rs
rustfs/src/auth.rs
rustfs/src/error.rs
rustfs/src/storage_api.rs
rustfs/src/app/storage_api.rs
rustfs/src/admin/router.rs
rustfs/src/app/metadata_route.rs
rustfs/src/server/http.rs
rustfs/src/server/layer.rs
rustfs/src/server/hybrid.rs
rustfs/src/protocols/client.rs
# e2e oracle: the test target that verifies S3 behaviour against s3s
crates/e2e_test/src/fake_s3_target/**
crates/e2e_test/src/**
+17
View File
@@ -0,0 +1,17 @@
# Exact baseline for the direct s3s footprint, consumed only by
# scripts/check_s3s_footprint.sh (rustfs/backlog#1677; rustfs/backlog#2734 T0.4).
#
# Each value is the exact measurement of the tree, not a ceiling. CI fails when a
# measured counter differs from its line here in either direction:
# - a larger measurement means new code widened the s3s surface: route it
# through the gateway abstractions instead; never raise a value here;
# - a smaller measurement means the PR shrank the surface: lower the matching
# value to the measured number in the same PR.
#
# files Rust files referencing s3s paths, repo-wide minus crates/e2e_test
# s3_error_lines lines holding s3_error!, same scope, minus those the script
# header's rule clears (rustfs_s3_types's macro); fails closed
# ecstore_files Rust files referencing s3s paths under crates/ecstore/src
files=65
s3_error_lines=289
ecstore_files=0
+63
View File
@@ -0,0 +1,63 @@
# Per-crate test-count baseline consumed by scripts/check_test_count_ratchet.sh
# (rustfs/backlog#2734 section 1 rule 3; T0.3 in rustfs/backlog#2738).
#
# One row per workspace member: <package>, tests, ignored, tab-separated.
# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
# attribute lines under the package directory; ignored counts #[ignore] and
# #[ignore = "..."]. Both columns are exact: CI fails when a crate's tests fall
# below or rise above its row, or when its ignored count changes. Raise a row in
# the PR that adds tests; lowering one is a reviewed statement that tests were
# intentionally removed. Regenerate with:
# scripts/check_test_count_ratchet.sh --update
e2e_test 956 20
rustfs 5955 64
rustfs-audit 70 0
rustfs-checksums 26 0
rustfs-common 36 0
rustfs-concurrency 53 0
rustfs-config 43 0
rustfs-connect-symbol-catalog 6 0
rustfs-credentials 24 0
rustfs-crypto 67 0
rustfs-data-usage 55 0
rustfs-ecstore 7081 1
rustfs-extension-schema 23 0
rustfs-filemeta 364 0
rustfs-heal 984 3
rustfs-heal-contracts 7 0
rustfs-iam 394 0
rustfs-io-core 49 0
rustfs-io-metrics 153 0
rustfs-keystone 29 0
rustfs-kms 715 25
rustfs-license 2 0
rustfs-lifecycle 155 0
rustfs-lock 138 0
rustfs-log-analyzer 90 0
rustfs-madmin 210 0
rustfs-notify 137 0
rustfs-object-capacity 92 0
rustfs-object-data-cache 140 0
rustfs-obs 379 0
rustfs-policy 725 0
rustfs-protocols 738 1
rustfs-protos 66 0
rustfs-replication 209 0
rustfs-rio 206 0
rustfs-rio-v2 34 6
rustfs-s3-client 69 0
rustfs-s3-ops 8 0
rustfs-s3-types 103 0
rustfs-s3select-api 207 0
rustfs-s3select-query 189 2
rustfs-scanner 1205 20
rustfs-scanner-metrics 87 0
rustfs-security-governance 20 0
rustfs-signer 31 0
rustfs-storage-api 34 0
rustfs-targets 603 26
rustfs-test-utils 1 0
rustfs-tls-runtime 43 0
rustfs-trusted-proxies 68 0
rustfs-utils 247 0
rustfs-zip 30 0
@@ -1748,7 +1748,7 @@
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_objects_count{job=~\"$job\", bucket=~\"$bucket\"})",
"expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
"legendFormat": "{{bucket}}",
"range": true,
"refId": "A"
@@ -1848,7 +1848,7 @@
"uid": "${datasource}"
},
"editorMode": "code",
"expr": "max by (job, bucket) (rustfs_cluster_usage_buckets_total_bytes{job=~\"$job\", bucket=~\"$bucket\"})",
"expr": "max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id=\"$storage_cluster\", job=~\"$job\", bucket=~\"$bucket\"})",
"legendFormat": "{{bucket}}",
"range": true,
"refId": "A"
+1
View File
@@ -61,6 +61,7 @@ scrape_configs:
otlp:
promote_resource_attributes:
- service.instance.id
- process.pid
- service.name
- service.namespace
- cloud.availability_zone
@@ -0,0 +1,48 @@
evaluation_interval: 1m
tests:
- name: bucket usage keeps clusters separate and deduplicates reporting nodes
interval: 1m
input_series:
- series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
values: '4096 1024'
- series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
values: '4096 1024'
- series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
values: '65536 65536'
- series: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
values: '0 0'
- series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a1"}
values: '3 1'
- series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="shared",service_instance_id="a2"}
values: '3 1'
- series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="b",job="rustfs",bucket="shared",service_instance_id="b1"}
values: '70 70'
- series: rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job="rustfs",bucket="empty",service_instance_id="a1"}
values: '0 0'
promql_expr_test:
- expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
eval_time: 1m
exp_samples:
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
value: 1024
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
value: 0
- expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_objects_count{rustfs_cluster_id="a",job=~"rustfs",bucket=~".*"})
eval_time: 1m
exp_samples:
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
value: 1
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
value: 0
- expr: max by (rustfs_cluster_id, job, bucket) (rustfs_cluster_usage_buckets_total_bytes)
eval_time: 1m
exp_samples:
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="shared"}'
value: 1024
- labels: '{rustfs_cluster_id="a",job="rustfs",bucket="empty"}'
value: 0
- labels: '{rustfs_cluster_id="b",job="rustfs",bucket="shared"}'
value: 65536
- expr: rustfs_cluster_usage_buckets_total_bytes{rustfs_cluster_id="a",bucket="unknown"}
eval_time: 1m
exp_samples: []
+5
View File
@@ -23,6 +23,11 @@ duplicate its rules here.
pass by fixing the cause, not by weakening the gate).
- If `shellcheck` is installed, actionlint also lints `run:` scripts; treat
those findings as part of the gate.
- Runner placement and billing rules live in `.github/workflows/README.md`.
A new job defaults to `ubuntu-latest` (free for this public repository);
a self-hosted `runs-on` needs one of the reasons listed there, stated in a
comment next to it, and a `rust-cache` key is never shared between the
hosted and self-hosted images.
## CI Alignment
-38
View File
@@ -1,38 +0,0 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: ''
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
**Smartphone (please complete the following information):**
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
**Additional context**
Add any other context about the problem here.
+110
View File
@@ -0,0 +1,110 @@
name: Bug report
description: Create a report to help us improve
body:
- type: markdown
attributes:
value: |
### Storage support requirements
**Supported:** local disks, SAN (Fibre Channel) attached volumes, or JBOD-mounted disks. We recommend formatting them with the **XFS** file system.
**Not supported:** NFS, SMB/Samba/CIFS, and other remote or shared filesystems as RustFS data directories. This also applies to the underlying host storage for Docker volumes and Kubernetes PersistentVolumes.
Reports that only reproduce on unsupported storage will be closed without investigation. Please reproduce the issue on supported storage before submitting a bug report.
See [Disk Planning and File System Selection](https://docs.rustfs.com/en/installation/linux/prerequisites-and-service) for details.
- type: checkboxes
id: storage-support
attributes:
label: Supported storage confirmation
options:
- label: I have reproduced this issue with all RustFS data directories on supported storage (local disks, SAN/Fibre Channel, or JBOD; XFS recommended), without NFS, SMB/Samba/CIFS, or other remote or shared filesystem mounts, including storage backing Docker volumes and Kubernetes PersistentVolumes.
required: true
- type: textarea
id: bug-description
attributes:
label: Describe the bug
description: A clear and concise description of what happened.
validations:
required: true
- type: textarea
id: reproduction-steps
attributes:
label: Steps to reproduce
description: Include the commands, configuration, or actions needed to reproduce the problem.
placeholder: |
1. Start RustFS with ...
2. Run ...
3. Observe ...
validations:
required: true
- type: textarea
id: expected-behavior
attributes:
label: Expected behavior
description: A clear and concise description of what you expected to happen.
validations:
required: true
- type: input
id: rustfs-version
attributes:
label: RustFS version
description: Provide the release version, container image tag, or commit hash.
validations:
required: true
- type: dropdown
id: deployment-method
attributes:
label: Deployment method
description: How are you running RustFS? Add any relevant details in Additional context.
options:
- Binary
- Docker / Docker Compose
- Kubernetes / Helm
- Built from source
- Other
- type: textarea
id: storage-mounts
attributes:
label: Data directory filesystems and mounts
description: |
Identify the filesystem type and mount source for every RustFS data directory.
On Linux, include output from `findmnt -T /path/to/data -o TARGET,SOURCE,FSTYPE,OPTIONS` for each data path; on other systems, provide equivalent mount information.
For Docker, identify the backing host path and filesystem, not just the container path or volume name.
For Kubernetes, identify the PV, StorageClass/CSI driver, and actual backing filesystem; a PVC name alone is not sufficient.
Remove credentials and other sensitive information before submitting.
placeholder: |
RustFS data paths:
Filesystem types and mount sources:
Mount command output:
Docker host paths or Kubernetes volume backing storage (if applicable):
validations:
required: true
- type: input
id: operating-system
attributes:
label: Server operating system and architecture
description: Include the operating system version and CPU architecture where RustFS is running.
placeholder: "For example: Ubuntu 24.04, x86_64"
validations:
required: true
- type: textarea
id: client-environment
attributes:
label: Client environment
description: Include the S3 client or SDK and version. For console issues, include the browser, operating system, and desktop or mobile device details.
- type: textarea
id: logs
attributes:
label: Relevant logs
description: Paste relevant log output. Remove credentials and other sensitive information before submitting.
render: shell
- type: textarea
id: screenshots
attributes:
label: Screenshots
description: If applicable, drag and drop screenshots here to help explain the problem.
- type: textarea
id: additional-context
attributes:
label: Additional context
description: Add any other context, such as deployment topology or relevant configuration. Remove credentials and other sensitive information before submitting.
+1
View File
@@ -0,0 +1 @@
blank_issues_enabled: false
-20
View File
@@ -1,20 +0,0 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
@@ -0,0 +1,38 @@
name: Feature request
description: Suggest an idea for this project
body:
- type: checkboxes
id: request-scope
attributes:
label: Request scope
description: |
RustFS data directories backed by NFS, SMB/Samba/CIFS, or other remote or shared filesystems are unsupported, including those behind Docker volumes or Kubernetes PersistentVolumes.
Troubleshooting requests for these deployments will be closed without investigation. Use this form for new or improved functionality.
options:
- label: This is a feature request and does not ask for troubleshooting an unsupported remote or shared filesystem deployment.
required: true
- type: textarea
id: problem-description
attributes:
label: Problem or use case
description: Describe the problem or use case that motivates this feature request.
placeholder: "For example: I am trying to ... but ..."
validations:
required: true
- type: textarea
id: proposed-solution
attributes:
label: Proposed solution
description: A clear and concise description of what you would like to happen.
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
description: Describe any alternative solutions or features you have considered.
- type: textarea
id: additional-context
attributes:
label: Additional context
description: Add any other context or drag and drop screenshots about the feature request here.
+10 -7
View File
@@ -19,7 +19,7 @@ runs:
using: composite
steps:
- name: Install quality tools
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
with:
tool: |
ripgrep@15.2.0
@@ -86,6 +86,10 @@ runs:
shell: bash
run: ./scripts/check_s3s_footprint.sh
- name: Check per-crate test-count ratchet
shell: bash
run: ./scripts/check_test_count_ratchet.sh
- name: Check cryptographic capability wording
shell: bash
run: ./scripts/check_fips_wording.sh
@@ -98,14 +102,13 @@ runs:
shell: bash
run: make script-tests
- name: Check object generation protocol model
shell: bash
run: python3 ./scripts/test_object_generation_protocol_model.py
- name: Check test wiring
shell: bash
run: |
python3 ./scripts/check_test_wiring.py --self-test
python3 ./scripts/check_scheduled_validation_freshness.py --self-test
python3 ./scripts/test_security_workflow.py
python3 ./scripts/test_nightly_candidate.py
python3 ./scripts/check_test_wiring.py
run: python3 ./scripts/check_test_wiring.py
- name: Check no planning docs committed
shell: bash
@@ -0,0 +1,39 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Set up Docker Compose
description: Install the verified Compose CLI used by native Linux x86_64 Connect acceptance
runs:
using: composite
steps:
- name: Install verified Docker Compose CLI plugin
shell: bash
run: |
set -euo pipefail
[[ $(uname -s) == Linux && $(uname -m) == x86_64 ]]
plugin_dir="${DOCKER_CONFIG:-$HOME/.docker}/cli-plugins"
mkdir -p "$plugin_dir"
compose_download=$(mktemp "${RUNNER_TEMP:?}/connect-compose.XXXXXX")
compose_staged=$(mktemp "$plugin_dir/.docker-compose.XXXXXX")
trap 'rm -f "$compose_download" "$compose_staged"' EXIT
curl --fail --location --retry 3 \
https://github.com/docker/compose/releases/download/v2.40.3/docker-compose-linux-x86_64 \
--output "$compose_download"
printf '%s %s\n' \
dba9d98e1ba5bfe11d88c99b9bd32fc4a0624a30fafe68eea34d61a3e42fd372 \
"$compose_download" | sha256sum --check --strict
install -m 0755 "$compose_download" "$compose_staged"
mv -f "$compose_staged" "$plugin_dir/docker-compose"
[[ $(docker compose version --short) == 2.40.3 ]]
+18 -2
View File
@@ -36,6 +36,16 @@ inputs:
description: "Install cross-compilation tools"
required: false
default: "false"
zig-version:
description: >-
Zig version for cargo-zigbuild builds. Pinned so the `<target>.<glibc>`
baseline semantics cannot drift with an untested zig release.
required: false
default: "0.17.0"
cargo-zigbuild-version:
description: "cargo-zigbuild version installed for cross builds."
required: false
default: "0.23.4"
target:
description: "Target architecture to add"
required: false
@@ -99,17 +109,23 @@ runs:
- name: Install Zig
if: inputs.install-cross-tools == 'true'
uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2
with:
version: ${{ inputs.zig-version }}
- name: Install cargo-zigbuild
if: inputs.install-cross-tools == 'true'
uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
with:
tool: cargo-zigbuild@${{ inputs.cargo-zigbuild-version }}
- name: Install cargo-nextest
if: inputs.install-test-tools == 'true'
uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
with:
tool: cargo-nextest@0.9.138
- name: Setup Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
with:
# false is rust-cache's own default. With true, cleanup.ts returns
# *before* pruning ~/.cargo/registry/src, and config.ts archives the
@@ -124,5 +140,5 @@ runs:
# normalises, so cargo fingerprints stay valid.
cache-all-crates: false
cache-on-failure: true
shared-key: ${{ inputs.cache-shared-key }}
shared-key: ${{ inputs.cache-shared-key }}${{ inputs.install-cross-tools == 'true' && format('-zig-{0}-zigbuild-{1}', inputs.zig-version, inputs.cargo-zigbuild-version) || '' }}
save-if: ${{ inputs.cache-save-if }}
+1 -1
View File
@@ -1,6 +1,6 @@
[
{ "workflow": ".github/workflows/audit.yml", "max_age_hours": 36 },
{ "workflow": ".github/workflows/build.yml", "max_age_hours": 192 },
{ "workflow": ".github/workflows/build.yml", "max_age_hours": 36 },
{ "workflow": ".github/workflows/ci.yml", "max_age_hours": 192 },
{ "workflow": ".github/workflows/coverage.yml", "max_age_hours": 192 },
{ "workflow": ".github/workflows/e2e-replication-nightly.yml", "max_age_hours": 36 },
+74
View File
@@ -0,0 +1,74 @@
# Workflow runners: what is billed and where a job belongs
**Use this when:** adding a job, changing a `runs-on`, moving a lane between runner pools, or deciding whether a new check is worth its runner cost.
**Source of truth:** `runs-on` in each workflow file for current placement; `.github/actionlint.yaml` for the declared self-hosted labels; the organisation's Settings → Billing → Usage page for what is actually charged; `docs/testing/ci-gates.md` for which checks block a merge (that does not change when a runner changes).
## Billing facts
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for by the hour whether or not a job is running well:
| Label | Paid by | What it is | Use it for |
|---|---|---|---|
| `ubuntu-latest` | nobody | GitHub-hosted, Ubuntu 24.04, 4 vCPU, 16 GB RAM, about 14 GB of free disk. Ships a C toolchain, `docker`, `gh`, `curl`, `wget`, `jq`, Python. Can `mount` tmpfs and loop devices; io_uring works | the default for every job |
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
| `sm-standard-4-arm` | RustFS | ARC pod, aarch64; reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds |
| `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | image builds that need `buildx` and a registry push |
| `smoke-testing`, `pf-testing` | RustFS | long-lived lab machines | the packaged functional chain and performance suites |
The self-hosted pods and the hosted image are both Ubuntu 24.04, so a binary built on one runs on the other; `ci.yml` builds the debug binary on `sm-standard-4` and runs it on both pools.
## Placement rule
A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason goes in a comment next to its `runs-on`:
1. **Disk.** It restores or produces the workspace `target/`: `cargo build` or `cargo test` with `--all-targets`, `cargo nextest run --all`, or any reader of the `ci-dev` / `ci-feat-*` caches. The workspace test build leaves about 64 GB in `target/`; a hosted runner has about 14 GB.
2. **Critical-path CPU.** It is a build that several downstream lanes wait on and would blow its timeout at 4 vCPU (`build-rustfs-debug-binary` gates the e2e and s3-tests lanes).
3. **Hardware.** Native aarch64, Docker-in-Docker, or the lab machines.
4. **Cache ownership.** It reads a `rust-cache` key that a self-hosted job writes (see the cache rule below).
Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
A hosted runner is also the only choice for a few things the pods cannot do: `mount -o loop` and `mount -t tmpfs` fail inside the ARC pods, and a container runtime cannot guarantee that `io_uring_setup` is allowed. `uring-integration` and `e2e-distributed` are on `ubuntu-latest` for that reason, not for cost.
## Cache rule
`Swatinem/rust-cache` builds its key from the OS, architecture, toolchain and lockfile. It does not include the runner label or image, so a cache written on an `sm-standard-4` pod and one written on `ubuntu-latest` would overwrite each other under the same `shared-key`.
- Every `shared-key` has exactly one writer. `cache-warm.yml` owns `ci-dev`, `ci-feat-rio`, `ci-feat-proto` and `ci-uring`; every other `./.github/actions/setup` call states `cache-save-if` explicitly, and `scripts/security/check_cache_save_if.sh` fails the build when one does not.
- All readers of a key run on the same image as its writer. Keys that live on hosted runners say so in their name (`ci-uring` is warmed on `ubuntu-latest`; `ci-e2e-distributed-hosted`, `e2e-upgrade-server-hosted`, `e2e-upgrade-tests-hosted`).
- Moving a job between pools therefore means renaming its key. The new key is empty until a run from `main` (a scheduled run, or a manual dispatch of that workflow from `main`) saves it; until then that lane does a cold build.
- `CARGO_BUILD_JOBS` and every other `CARGO*`, `CC*`, `CFLAGS*`, `CXX*`, `CMAKE*` and `RUST*` variable is hashed into the key. `ci.yml` sets `CARGO_BUILD_JOBS` at step level for that reason; promoting it to job level rotates the key for every lane that shares it.
## Where self-hosted minutes leak
These are the patterns that have cost real money here without validating anything more:
- An `if: always()` summary or alert job on a self-hosted label inside a `workflow_run`-triggered workflow. The upstream workflow completes many times a day, the real jobs skip, and the summary still starts a pod each time. Gate it on the upstream job having run, and keep it on `ubuntu-latest`.
- Per-push workflows on `main`. A merge triggers the full `ci.yml` matrix, `cache-warm.yml` and the `build.yml` development build; with tens of merges a day that is as much self-hosted time as all pull requests together. Anything that does not verify the merged tree itself belongs on a schedule or behind a `paths` filter.
- `apt-get install` of tools the hosted image already ships. It is only needed on the minimal pod images.
- A throttled `CARGO_BUILD_JOBS` on a 14-CPU pod. The limit exists for link-storm reasons documented in `ci.yml`; measure before raising it, and measure before lowering it further.
- Tiny jobs on ARC pods. Each one pays roughly a minute of pod start-up before its first step; a compile-free job that takes ten seconds costs more to schedule than to run.
## Measuring a run
To see which pool each job of a run used and for how long:
```bash
gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
```
Only rows whose label starts with `sm-`, `dind-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
## Checklist for a new or moved job
- `runs-on` is a literal label. A new self-hosted label is declared in `.github/actionlint.yaml` in the same change (`.github/AGENTS.md`).
- The job declares `timeout-minutes` (`scripts/security/check_job_timeouts.sh`).
- `actionlint` reports nothing.
- A self-hosted `runs-on` carries a comment naming its reason from the placement rule.
- Every `./.github/actions/setup` call states `cache-save-if`, and a key read from one pool is not written from the other.
- Moving a job does not change its check name, so the `main` ruleset and `docs/testing/ci-gates.md` stay as they are; renaming it does, and then that document's change checklist applies.
@@ -1,61 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Architecture Migration Rules
on:
pull_request:
types: [ opened, synchronize, reopened, closed ]
branches: [ main ]
paths:
- "ARCHITECTURE.md"
- "docs/architecture/**"
- "scripts/check_architecture_migration_rules.sh"
- ".github/workflows/architecture-migration-rules.yml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
cancel-closed-pr-runs:
name: Cancel Closed PR Runs
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Explain cancellation run
run: echo "PR closed; this run only cancels older runs in the same concurrency group."
architecture-migration-rules:
name: Architecture Migration Rules
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
- name: Install ripgrep
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
with:
tool: ripgrep@15.2.0
- name: Check architecture migration rules
run: ./scripts/check_architecture_migration_rules.sh
+9 -9
View File
@@ -88,12 +88,12 @@ jobs:
# database lookup with a second ignore list to maintain.
cargo-deny:
name: Cargo Deny
if: github.event_name != 'pull_request' || github.event.action != 'closed'
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -112,7 +112,7 @@ jobs:
# default is now "false", but state it explicitly — see
# scripts/security/check_cache_save_if.sh.
- name: Setup Rust cache
uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2
with:
# Same reasoning as the setup composite: true archives every
# dependency's unpacked source tree.
@@ -122,7 +122,7 @@ jobs:
save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install cargo-deny
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
with:
tool: cargo-deny
@@ -131,12 +131,12 @@ jobs:
workflow-pin-report:
name: Workflow Pin Report
if: github.event_name != 'pull_request' || github.event.action != 'closed'
if: github.repository == 'rustfs/rustfs' && (github.event_name != 'pull_request' || github.event.action != 'closed')
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -177,7 +177,7 @@ jobs:
pull-requests: write
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -203,14 +203,14 @@ jobs:
# workflow produces, and until now it was only visible to whoever happened to
# open the Actions tab. Same ci-8 mechanism coverage.yml and
# e2e-replication-nightly.yml already use.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+173 -140
View File
@@ -22,12 +22,16 @@
# 3. Trigger docker.yml to build and push images using the uploaded binaries
#
# Platform scope:
# - Pushes to main (development builds) build the Linux targets only — they
# feed the dev download channel and Docker dev images. Tags, the weekly
# schedule and manual dispatch build the full platform matrix.
# - Development builds come from the nightly schedule and manual dispatch and
# cover the full platform matrix; they feed the dev download channel
# (artifacts/rustfs/dev on R2, the *-main-latest packages). Tags build the
# release matrix. Main pushes do not build: at the usual merge rate most
# per-merge builds were cancelled by the next merge before publishing
# anything, and the dev channel does not need per-merge granularity.
#
# Manual Parameters:
# - build_docker: Build and push Docker images (default: true)
# - publish: Publish binaries/releases for manual runs (default: false)
# - build_docker: Advisory only; manual runs do not trigger Docker publishing
# - platforms: Comma-separated platform IDs or 'all' (default: all)
name: Build and Release
@@ -35,32 +39,20 @@ name: Build and Release
on:
push:
tags: [ "*.*.*" ]
branches: [ main ]
paths-ignore:
- "**.md"
- "**.txt"
- ".github/**"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "LICENSE*"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".gitignore"
- ".dockerignore"
- "flake.lock"
schedule:
- cron: "13 1 * * 0" # Weekly on Sunday 01:13 UTC
- cron: "13 1 * * *" # Nightly 01:13 UTC
workflow_dispatch:
inputs:
publish:
description: "Publish to download channels and GitHub Releases (false keeps only workflow artifacts)"
required: false
default: false
type: boolean
build_docker:
# Advisory only. docker.yml triggers on workflow_run and its job-level
# condition requires the triggering event to be a tag push, so a manual
# dispatch of this workflow never produces images regardless of this
# value. Kept because the summary step reports it; wiring it up would
# mean teaching docker.yml's version parser a second event shape.
# value. Publishing remains controlled by the explicit publish input.
description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
required: false
default: true
@@ -92,13 +84,14 @@ jobs:
timeout-minutes: 30
outputs:
should_build: ${{ steps.check.outputs.should_build }}
should_publish: ${{ github.event_name != 'workflow_dispatch' || inputs.publish == true }}
build_type: ${{ steps.check.outputs.build_type }}
version: ${{ steps.check.outputs.version }}
short_sha: ${{ steps.check.outputs.short_sha }}
is_prerelease: ${{ steps.check.outputs.is_prerelease }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -138,12 +131,6 @@ jobs:
build_type="release"
echo "📦 Release build detected: $tag_name"
fi
elif [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
# Main branch push - development build
should_build=true
build_type="development"
version="dev-${short_sha}"
echo "🛠️ Development build detected"
elif [[ "${{ github.event_name }}" == "schedule" ]] || \
[[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
# Scheduled or manual build
@@ -194,25 +181,14 @@ jobs:
selected="all"
fi
# Per-merge development builds only feed the dev download channel
# and the Docker dev images, which consume the Linux binaries; at
# the usual merge cadence most per-merge builds are cancelled by the
# next merge anyway. macOS and Windows stay covered by tag builds,
# the weekly scheduled build and manual dispatch.
if [[ "${selected}" == "all" \
&& "${{ github.event_name }}" == "push" \
&& "${{ needs.build-check.outputs.build_type }}" == "development" ]]; then
selected="linux-x86_64-musl,linux-aarch64-musl,linux-x86_64-gnu,linux-aarch64-gnu"
echo "Development (main push) build: restricting to Linux targets"
fi
all='{"include":[
{"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":""},
{"target_id":"linux-aarch64-musl","os":"sm-standard-4","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":"","features":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":"","features":"pyroscope"},
{"target_id":"linux-aarch64-gnu","os":"sm-standard-4","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope"},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope"},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":""}
{"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
]}'
if [[ "${selected}" == "all" ]]; then
@@ -240,7 +216,8 @@ jobs:
echo "Selected platforms: ${selected}"
build-rustfs:
name: Build RustFS
# Acceptance workflows select artifacts by this stable platform name.
name: Build RustFS (${{ matrix.target_id }})
needs: [ build-check, prepare-platform-matrix ]
if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
runs-on: ${{ matrix.os }}
@@ -257,7 +234,7 @@ jobs:
matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0
@@ -267,7 +244,7 @@ jobs:
with:
rust-version: stable
target: ${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}
# main only. A cache saved on refs/tags/X is scoped to that tag: no
# other tag, no main run and no PR can restore it, so every release
# cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
@@ -402,29 +379,83 @@ jobs:
- name: Build RustFS
shell: bash
env:
BUILD_TARGET: ${{ matrix.target }}
BUILD_OS: ${{ matrix.platform }}-${{ startsWith(matrix.target, 'aarch64-') && 'aarch64' || 'x86_64' }}
run: |
# Force rebuild by touching build.rs
touch rustfs/build.rs
BINARY_NAMES=(rustfs)
if [[ "${{ matrix.platform }}" == "linux" ]]; then
BINARY_NAMES+=(rustfs-cli)
fi
FEATURE_ARGS=()
if [[ -n "${{ matrix.features }}" ]]; then
FEATURE_ARGS=(--features "${{ matrix.features }}")
fi
# Build one binary at a time so release LTO links cannot exhaust a
# self-hosted runner by running concurrently.
for binary in "${BINARY_NAMES[@]}"; do
if [[ "${{ matrix.cross }}" == "true" ]]; then
# All cross targets in the matrix are Linux; zigbuild handles them.
cargo zigbuild --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" -p rustfs --bin "$binary"
else
cargo build --release --target ${{ matrix.target }} "${FEATURE_ARGS[@]}" -p rustfs --bin "$binary"
fi
done
# gnu targets pin an explicit glibc baseline (`<target>.<glibc>` for
# cargo-zigbuild) so release binaries link no symbols newer than it
# and run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
TARGET_ARG="$BUILD_TARGET"
if [[ -n "${{ matrix.glibc }}" ]]; then
TARGET_ARG="${TARGET_ARG}.${{ matrix.glibc }}"
fi
PROFILE_ARGS=()
if [[ "$BUILD_TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
# The in-process profiler resolves names from the final executable.
# Retain function symbols without enabling DWARF source information.
PROFILE_ARGS=(--config 'profile.release.strip="none"')
fi
if [[ "${{ matrix.cross }}" == "true" ]]; then
cargo zigbuild --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
else
cargo build --release --locked --target "$TARGET_ARG" "${FEATURE_ARGS[@]}" "${PROFILE_ARGS[@]}" -p rustfs --bin rustfs
fi
# All release targets use same-architecture runners, including GNU
# builds that use Zig only to select the glibc ABI floor.
bin="target/${{ matrix.target }}/release/rustfs"
if [[ -f "${bin}.exe" ]]; then
bin="${bin}.exe"
fi
version_output=$("$bin" --version)
if ! grep -Fxq "build target : ${BUILD_TARGET}" <<< "$version_output" || \
! grep -Fxq "build os : ${BUILD_OS}" <<< "$version_output" || \
! grep -Fxq "rust channel : stable-${BUILD_TARGET}" <<< "$version_output"; then
echo "release binary reports an unexpected compile target" >&2
exit 1
fi
if [[ -n "${{ matrix.glibc }}" ]]; then
python3 scripts/release/check_glibc.py "${{ matrix.glibc }}" \
"target/${{ matrix.target }}/release/rustfs"
fi
- name: Verify macOS binary has no dynamic liblzma dependency
if: matrix.platform == 'macos'
shell: bash
run: |
binary="target/${{ matrix.target }}/release/rustfs"
if otool -L "$binary" | grep -E 'liblzma.*\.dylib'; then
echo "macOS release binary must not depend on a dynamic liblzma library" >&2
exit 1
fi
"$binary" --version
- name: Generate the CPU symbol catalog
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
shell: bash
run: |
cargo metadata --locked --format-version 1 --filter-platform '${{ matrix.target }}' \
--features rustfs/pyroscope > "$RUNNER_TEMP/cpu-catalog-metadata.json"
features=$(jq -ce '
(.packages[] | select(.name == "rustfs") | .id) as $id |
.resolve.nodes[] | select(.id == $id) | .features | map(select(. != "default")) | sort
' "$RUNNER_TEMP/cpu-catalog-metadata.json")
release_dir="target/${{ matrix.target }}/release"
# These task-owned outputs may remain in a restored target cache.
rm -f "$release_dir/rustfs.cpu-symbol-catalog.json" "$release_dir/rustfs.cpu-symbol-catalog.sha256"
cargo run --locked -p rustfs-connect-symbol-catalog -- \
"$release_dir/rustfs" "$(git rev-parse HEAD)" '${{ matrix.target }}' "$features" \
"$release_dir/rustfs.cpu-symbol-catalog.json" > "$release_dir/rustfs.cpu-symbol-catalog.sha256"
- name: Create release package
id: package
@@ -512,8 +543,9 @@ jobs:
fi
BINARY_FILES=("$BINARY_NAME")
if [[ "${{ matrix.platform }}" == "linux" ]]; then
BINARY_FILES+=("rustfs-cli")
if [[ "$TARGET" == *-linux-gnu && "${{ matrix.features }}" == "pyroscope" ]]; then
BINARY_FILES+=(rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256)
fi
# Verify the binaries exist before packaging
@@ -583,7 +615,7 @@ jobs:
LATEST_FILES="$LATEST_FILE"
fi
elif [[ "$BUILD_TYPE" == "development" ]]; then
# Development builds (only main branch triggers development builds)
# Development builds (nightly schedule and manual dispatch)
# Create main-latest version filename
# Convert from rustfs-linux-x86_64-dev-abc123 to rustfs-linux-x86_64-main-latest
MAIN_LATEST_FILE="${PACKAGE_BASENAME}-main-latest.zip"
@@ -625,7 +657,7 @@ jobs:
echo "🔧 Build type: ${BUILD_TYPE}"
echo "📊 Version: ${VERSION}"
- name: Verify packaged Linux CLI
- name: Verify packaged Linux binary
if: matrix.platform == 'linux'
shell: bash
run: |
@@ -633,22 +665,29 @@ jobs:
package_file="${{ steps.package.outputs.package_file }}"
archive_members="$(unzip -Z1 "$package_file" | sort)"
expected_members=$'rustfs\nrustfs-cli'
expected_members='rustfs'
target="${{ matrix.target }}"
features="${{ matrix.features }}"
if [[ "$target" == *-linux-gnu && "$features" == "pyroscope" ]]; then
expected_members+=$'\nrustfs.cpu-symbol-catalog.json\nrustfs.cpu-symbol-catalog.sha256'
fi
if [[ "$archive_members" != "$expected_members" ]]; then
echo "Linux package must contain exactly rustfs and rustfs-cli" >&2
echo "Linux package does not contain the expected files" >&2
printf '%s\n' "$archive_members" >&2
exit 1
fi
if [[ "${{ matrix.target }}" == "x86_64-unknown-linux-gnu" ]]; then
cli_dir="$(mktemp -d)"
trap 'rm -rf "$cli_dir"' EXIT
unzip -q "$package_file" rustfs-cli -d "$cli_dir"
"$cli_dir/rustfs-cli" --help >/dev/null
fi
# musl targets build natively; the gnu legs are same-arch
# cargo-zigbuild builds pinned to the release glibc baseline. Either
# way the binaries run on their runner, so executing them here
# proves each artifact matches its runner architecture.
binary_dir="$(mktemp -d)"
trap 'rm -rf "$binary_dir"' EXIT
unzip -q "$package_file" rustfs -d "$binary_dir"
"$binary_dir/rustfs" --version
- name: Verify packaged console
if: matrix.target == 'x86_64-unknown-linux-gnu'
if: matrix.target == 'x86_64-unknown-linux-gnu' || matrix.target == 'aarch64-unknown-linux-gnu'
shell: bash
run: |
set -euo pipefail
@@ -702,14 +741,14 @@ jobs:
exit 1
- name: Upload to GitHub artifacts
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ steps.package.outputs.package_name }}
path: "rustfs-*.zip"
retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}
- name: Upload to Cloudflare R2
if: env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
if: needs.build-check.outputs.should_publish == 'true' && env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
@@ -726,12 +765,33 @@ jobs:
fi
# The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted
# images did). Install it on demand so R2 uploads survive a fresh runner
# instead of hard-failing here.
# images did). Install the official CLI bundle on demand so R2 uploads
# survive a fresh runner without relying on a distro package.
if ! command -v aws >/dev/null 2>&1; then
echo "aws CLI not found on runner; installing..."
if command -v apt-get >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
aws_arch="$(uname -m)"
case "$aws_arch" in
x86_64) aws_arch="x86_64" ;;
aarch64|arm64) aws_arch="aarch64" ;;
*)
echo "❌ Unsupported Linux architecture for AWS CLI: $aws_arch"
exit 1
;;
esac
aws_install_root="${RUNNER_TEMP:-$PWD}/aws-cli"
rm -rf "$aws_install_root"
mkdir -p "$aws_install_root"
curl --fail --location --retry 3 \
"https://awscli.amazonaws.com/awscli-exe-linux-${aws_arch}.zip" \
--output "$aws_install_root/awscliv2.zip"
unzip -q "$aws_install_root/awscliv2.zip" -d "$aws_install_root"
"$aws_install_root/aws/install" \
--install-dir "$aws_install_root/aws-cli" \
--bin-dir "$aws_install_root/bin"
export PATH="$aws_install_root/bin:$PATH"
aws --version
elif command -v brew >/dev/null 2>&1; then
brew install awscli
else
@@ -776,62 +836,35 @@ jobs:
- name: Build completion summary
shell: bash
env:
# dispatch input via env: free-form text must not be pasted into the
# script for bash to evaluate.
INPUT_BUILD_DOCKER: ${{ github.event.inputs.build_docker }}
SHOULD_PUBLISH: ${{ needs.build-check.outputs.should_publish }}
BUILD_TYPE: ${{ needs.build-check.outputs.build_type }}
VERSION: ${{ needs.build-check.outputs.version }}
BUILD_STATUS: ${{ needs.build-rustfs.result }}
EVENT_NAME: ${{ github.event_name }}
run: |
BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
VERSION="${{ needs.build-check.outputs.version }}"
echo "🎉 Build completed successfully!"
echo "📦 Build type: $BUILD_TYPE"
echo "🔢 Version: $VERSION"
echo ""
# Check build status
BUILD_STATUS="${{ needs.build-rustfs.result }}"
echo "📊 Build Results:"
echo " 📦 All platforms: $BUILD_STATUS"
echo ""
case "$BUILD_TYPE" in
"preview")
echo "🔍 Preview artifacts are published in a GitHub prerelease"
echo "⏭️ Preview releases do not update latest channels"
;;
"development")
echo "🛠️ Development build artifacts have been uploaded to OSS dev directory"
echo "⚠️ This is a development build - not suitable for production use"
;;
"release")
echo "🚀 Release build artifacts have been uploaded to OSS release directory"
echo "✅ This build is ready for production use"
echo "🏷️ GitHub Release will be created in this workflow"
;;
"prerelease")
echo "🧪 Prerelease build artifacts have been uploaded to OSS release directory"
echo "⚠️ This is a prerelease build - use with caution"
echo "🏷️ GitHub Release will be created in this workflow"
;;
esac
echo ""
echo "🐳 Docker Images:"
if [[ "$INPUT_BUILD_DOCKER" == "false" ]]; then
echo "⏭️ Docker image build was skipped (binary only build)"
echo "Build type: $BUILD_TYPE"
echo "Version: $VERSION"
echo "Build result: $BUILD_STATUS"
if [[ "$SHOULD_PUBLISH" != "true" ]]; then
echo "Artifact-only run: no download channels or GitHub Releases are updated."
echo "Successful platform packages are available as workflow artifacts."
elif [[ "$BUILD_STATUS" == "success" ]]; then
echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
echo "Publishing enabled; check the upload and release jobs for their results."
else
echo "❌ Docker image build will be skipped due to build failure"
echo "Build incomplete; check individual platform jobs for any completed uploads."
fi
if [[ "$EVENT_NAME" == "push" && "$GITHUB_REF" == refs/tags/* && "$BUILD_STATUS" == "success" ]]; then
echo "Docker publishing is eligible through the downstream workflow."
else
echo "This run does not trigger Docker publishing."
fi
# Create GitHub Release for every valid release tag, including previews
create-release:
name: Create GitHub Release
needs: [ build-check, build-rustfs ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
@@ -840,7 +873,7 @@ jobs:
release_url: ${{ steps.create.outputs.release_url }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0
@@ -891,20 +924,20 @@ jobs:
upload-release-assets:
name: Upload Release Assets
needs: [ build-check, build-rustfs, create-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
@@ -985,7 +1018,7 @@ jobs:
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
@@ -1045,8 +1078,8 @@ jobs:
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
@@ -1083,8 +1116,8 @@ jobs:
cleanup-preview-releases:
name: Cleanup Preview Releases
needs: [ build-check, publish-release ]
if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
@@ -1138,7 +1171,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+42 -33
View File
@@ -15,16 +15,16 @@
# Sole writer of the Rust dependency caches that ci.yml restores.
#
# Why this is a separate workflow rather than steps inside ci.yml: ci.yml's
# concurrency group cancels in-progress runs on main pushes, and merges land far
# faster than its 70-minute pipeline. Measured over 15 consecutive main pushes:
# concurrency group originally cancelled in-progress main runs, while merges
# landed faster than its 70-minute pipeline. Over 15 consecutive main pushes:
# 12 cancelled, 2 failed, 0 succeeded. A cancelled run never reaches
# Swatinem/rust-cache's post step (cache-on-failure does not cover cancellation),
# so the writer lanes were saving nothing and every PR paid a cold restore —
# 11.8-20.9 minutes of "Setup Rust environment" against 0.7-3.4 warm.
#
# Splitting cache writing out of the test pipeline lets ci.yml keep cancelling
# superseded runs (which is correct — nobody needs test results for a commit
# that is already three merges behind) while the caches still get written.
# Keep cache writing separate from validation: only these jobs publish the
# complete feature closure for each key. Main validation now also finishes its
# running baseline, while superseded PR attempts can still be cancelled.
#
# The group below deliberately does NOT cancel in progress; see the comment on
# it for how that bounds concurrency and why it is scoped by event.
@@ -42,26 +42,23 @@
name: Cache Warm
on:
schedule:
# Every six hours; the 22:41 UTC run lands before ci.yml's nightly 00:11
# UTC matrix. A reader restores the newest entry under its key and
# rebuilds whatever its own diff changed, so a cache up to six hours old
# costs a PR the handful of crates merged since. Warming after every merge
# (fourteen completed runs a day, about two sm-standard-4 hours each)
# spent far more than that to avoid it.
- cron: "41 4,10,16,22 * * *"
push:
branches: [ main ]
# Mirrors ci.yml's push paths-ignore: if a commit cannot change what ci.yml
# compiles, it cannot change what ci.yml needs restored either.
paths-ignore:
- "**.md"
- "docs/**"
- "deploy/**"
- "scripts/dev_*.sh"
- "scripts/probe.sh"
- "LICENSE*"
- ".gitignore"
- ".dockerignore"
- "README*"
- "**/*.png"
- "**/*.jpg"
- "**/*.svg"
- ".github/workflows/build.yml"
- ".github/workflows/docker.yml"
- ".github/workflows/audit.yml"
# A toolchain or lockfile change invalidates most of every key; those
# re-warm at once instead of waiting for the next slot.
paths:
- "Cargo.lock"
- "rust-toolchain.toml"
- ".github/actions/setup/**"
- ".github/workflows/cache-warm.yml"
workflow_dispatch:
inputs:
emit_timings:
@@ -75,7 +72,7 @@ on:
permissions:
contents: read
# Scoped by event. A push run and a dispatch run do not compete: GitHub keeps
# Scoped by event. Scheduled, push and dispatch runs do not compete: GitHub keeps
# one running plus one pending per group, so with a single shared group a
# manually dispatched run was displaced as pending by the next merge and
# cancelled — observed three times in a row, which made the --timings gate in
@@ -99,7 +96,7 @@ env:
RUST_BACKTRACE: 1
jobs:
# Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
# Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
# e2e-tests, e2e-full.
warm-ci-dev:
name: Warm ci-dev
@@ -109,7 +106,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -141,7 +138,7 @@ jobs:
- name: Upload cargo timings report
if: inputs.emit_timings
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: cargo-timings-ci-dev
path: target/cargo-timings/
@@ -162,6 +159,17 @@ jobs:
cargo build --workspace --all-targets
cargo build -p rustfs --bins --features e2e-test-hooks
# ci.yml runs Clippy on every pull request. Clippy's check-mode artifacts
# are separate from the build above, so warm them with the exact
# invocation ci.yml uses; a pull request then re-checks only what its diff
# touches. A lint failure on main must not stop the warm-up: the cache is
# still saved (cache-on-failure), and the failure belongs to ci.yml.
- name: Warm Clippy check artifacts
run: |
if ! cargo clippy --all-targets -- -D warnings; then
echo "::warning::cargo clippy reports findings on this commit; check artifacts are still cached"
fi
# Runs before rust-cache's post step, so these are the sizes it is about
# to archive. Reported so the cache-all-crates decision stays evidence-led:
# registry/src is what that flag prunes, registry/cache is what the pruned
@@ -185,9 +193,10 @@ jobs:
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
# Readers: test-and-lint-rio-v2 (per-PR), build-rustfs-debug-binary-rio-v2
# (weekly schedule / manual dispatch only — dormant rio-v2 variant, see
# rustfs/backlog#1835 and docs/architecture/minio-file-format-compat.md).
# Readers: test-and-lint-rio-v2 (nightly schedule, merge group, dispatch),
# build-rustfs-debug-binary-rio-v2 (nightly schedule / manual dispatch only —
# dormant rio-v2 variant, see rustfs/backlog#1835 and
# docs/architecture/minio-file-format-compat.md).
# The second build below stays despite the reduced cadence: it warms the
# rio-v2,e2e-test-hooks feature resolution the scheduled build restores,
# which keeps that lane inside its 30-minute timeout.
@@ -199,7 +208,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -230,7 +239,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -260,7 +269,7 @@ jobs:
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
+348 -258
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -66,7 +66,7 @@ jobs:
steps:
- name: Report CLA result for merge queue
if: github.event_name == 'merge_group'
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
await github.rest.checks.create({
@@ -0,0 +1,206 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Connect diagnostic scheduler receipt acceptance
on:
workflow_dispatch:
inputs:
build_run_id:
description: Main-branch Build and Release run ID
required: true
type: string
artifact_id:
description: Linux x86_64 GNU artifact from that run
required: true
type: string
source_sha:
description: Exact RustFS source commit
required: true
type: string
artifact_digest:
description: GitHub artifact digest including sha256 prefix
required: true
type: string
binary_sha256:
description: Independently verified rustfs binary SHA-256
required: true
type: string
connect_sha:
description: Exact Connect scheduler acceptance harness commit
required: true
type: string
permissions:
actions: read
contents: read
jobs:
diagnostic-schedule:
name: Verify production scheduler receipts over mTLS
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/connect
path: connect-harness
persist-credentials: false
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Verify official source and artifact identity
shell: bash
env:
GH_TOKEN: ${{ github.token }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
SOURCE_SHA: ${{ inputs.source_sha }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
run: |
set -euo pipefail
[[ "$BUILD_RUN_ID" =~ ^[0-9]+$ && "$ARTIFACT_ID" =~ ^[0-9]+$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
[[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
jq -e --arg source "$SOURCE_SHA" --arg repository "$GITHUB_REPOSITORY" '
.head_sha == $source and .head_branch == "main"
and .head_repository.full_name == $repository
and .name == "Build and Release" and .path == ".github/workflows/build.yml"
and .status == "completed" and .conclusion == "success"
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
.workflow_run.id == $run and .workflow_run.head_sha == $source
and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
and .digest == $digest and .expired == false
and (.expires_at | fromdateiso8601) > now
and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
' <<<"$artifact" >/dev/null
gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
printf '%s official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
- name: Extract the verified official binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
run: |
set -euo pipefail
python3 - <<'PY'
import io
import os
import pathlib
import stat
import zipfile
def regular(entry):
mode = entry.external_attr >> 16
return not entry.is_dir() and (stat.S_IFMT(mode) in (0, stat.S_IFREG))
package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
with zipfile.ZipFile('official-artifact.zip') as outer:
entries = outer.infolist()
assert 1 <= len(entries) <= 16
assert len({item.filename for item in entries}) == len(entries)
assert all(regular(item) and pathlib.PurePosixPath(item.filename).name == item.filename for item in entries)
assert sum(item.file_size for item in entries) <= 2147483648
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
assert {item.filename for item in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
assert all(regular(item) and 0 < item.file_size <= 1073741824 for item in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
import shutil
shutil.copyfileobj(source, destination)
PY
chmod 0755 binary/rustfs
printf '%s binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
file binary/rustfs | grep -Eq 'ELF 64-bit LSB.*x86-64'
- name: Run the real scheduler and receipt delivery scenario
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
git -C connect-harness diff --exit-code
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
[[ $(git -C connect-harness diff --name-only) == tests/e2e/connected/rustfs-ref ]]
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/diagnostic-schedule-evidence.test.sh)
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_DIAGNOSTIC_SCHEDULE_EVIDENCE="$GITHUB_WORKSPACE/diagnostic-schedule-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=diagnostic-schedule
- name: Bind and validate sanitized evidence
shell: bash
env:
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
jq -e --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" '
.provenance.connectSha == $connect and .provenance.sourceSha == $source
and .provenance.binarySha256 == $binary
' diagnostic-schedule-evidence.json >/dev/null
jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
--arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" '
.provenance += {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest}
' diagnostic-schedule-evidence.json > diagnostic-schedule-evidence.bound.json
mv diagnostic-schedule-evidence.bound.json diagnostic-schedule-evidence.json
node connect-harness/tests/connectivity/verify-diagnostic-schedule-evidence.mjs diagnostic-schedule-evidence.json --bound
- name: Upload verified receipt evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: connect-diagnostic-schedule-evidence-${{ github.run_id }}
path: diagnostic-schedule-evidence.json
retention-days: 14
if-no-files-found: error
@@ -0,0 +1,244 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Connect bounded health service acceptance
on:
workflow_dispatch:
inputs:
build_run_id:
description: Successful main-branch Build and Release run ID
required: true
type: string
artifact_id:
description: Linux x86_64 GNU artifact from that run
required: true
type: string
source_sha:
description: Exact RustFS source commit
required: true
type: string
artifact_digest:
description: GitHub artifact digest including sha256 prefix
required: true
type: string
binary_sha256:
description: Independently verified rustfs binary SHA-256
required: true
type: string
connect_sha:
description: Exact Connect health service acceptance harness commit
required: true
type: string
permissions:
actions: read
contents: read
jobs:
health-service:
name: Verify bounded health service over mTLS
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/connect
path: connect-harness
persist-credentials: false
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Verify official source and artifact identity
shell: bash
env:
GH_TOKEN: ${{ github.token }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
SOURCE_SHA: ${{ inputs.source_sha }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
run: |
set -euo pipefail
[[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
[[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
jq -e --arg source "$SOURCE_SHA" '
.head_sha == $source and .head_branch == "main"
and .head_repository.full_name == "rustfs/rustfs"
and .name == "Build and Release" and .path == ".github/workflows/build.yml"
and .status == "completed" and .conclusion == "success"
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
.workflow_run.id == $run and .workflow_run.head_sha == $source
and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
and .digest == $digest and .expired == false
and (.expires_at | fromdateiso8601) > now
and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
' <<<"$artifact" >/dev/null
jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
[[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
printf '%s official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
- name: Extract only the verified official service binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
run: |
set -euo pipefail
python3 - <<'PY'
import io
import os
import pathlib
import shutil
import stat
import zipfile
def regular(entry):
mode = entry.external_attr >> 16
return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
with zipfile.ZipFile('official-artifact.zip') as outer:
entries = outer.infolist()
assert 1 <= len(entries) <= 16
assert len({entry.filename for entry in entries}) == len(entries)
assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
assert sum(entry.file_size for entry in entries) <= 2147483648
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
assert {entry.filename for entry in entries} == {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'} and len(entries) == 3
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
header = source.read(20)
assert header[:7] == b'\x7fELF\x02\x01\x01' and header[18:20] == b'\x3e\x00', 'Expected a little-endian ELF64 x86-64 binary'
destination.write(header)
shutil.copyfileobj(source, destination)
PY
chmod 0755 binary/rustfs
printf '%s binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
- name: Run the real health service job
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
git -C connect-harness diff --exit-code
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
changed=$(git -C connect-harness diff --name-only)
[[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/health-service-evidence.test.sh)
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_HEALTH_SERVICE_EVIDENCE="$GITHUB_WORKSPACE/health-service-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=health-service
- name: Bind and validate sanitized evidence
shell: bash
env:
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
jq -e --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" '
.provenance.connectSha == $connect and .provenance.sourceSha == $source
and .provenance.binarySha256 == $binary
' health-service-evidence.json >/dev/null
jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
--arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
--arg expiry "$(cat artifact-expires-at)" '
.provenance += {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry}
' health-service-evidence.json > health-service-evidence.bound.json
mv health-service-evidence.bound.json health-service-evidence.json
node connect-harness/tests/connectivity/health-service-evidence.mjs health-service-evidence.json --bound
- name: Upload only verified sanitized evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: connect-health-service-evidence-${{ github.run_id }}
path: health-service-evidence.json
retention-days: 14
if-no-files-found: error
@@ -49,23 +49,23 @@ permissions:
jobs:
performance-drive:
name: Verify native Linux x86_64 performance.drive service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -73,10 +73,18 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +115,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +130,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -154,7 +162,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PERFORMANCE_DRIVE_EVIDENCE="$GITHUB_WORKSPACE/performance-drive-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=performance-drive
@@ -183,7 +191,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('performance-drive-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-performance-drive-evidence-${{ github.run_id }}
path: |
@@ -49,23 +49,23 @@ permissions:
jobs:
performance-network:
name: Verify native Linux x86_64 performance.network service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -73,10 +73,18 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +115,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +130,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -154,7 +162,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PERFORMANCE_NETWORK_EVIDENCE="$GITHUB_WORKSPACE/performance-network-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=performance-network
@@ -183,7 +191,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('performance-network-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-performance-network-evidence-${{ github.run_id }}
path: |
@@ -37,6 +37,10 @@ on:
description: Expected rustfs binary SHA-256
required: true
type: string
catalog_sha256:
description: Independently reviewed release CPU catalog SHA-256
required: true
type: string
connect_sha:
description: Exact 40-character RustFS Connect harness commit
required: true
@@ -49,23 +53,23 @@ permissions:
jobs:
profile-cpu:
name: Verify native Linux x86_64 profile.cpu service job
runs-on: ubuntu-latest
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -73,10 +77,18 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -101,20 +113,27 @@ jobs:
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
github_api() {
curl --fail --silent --show-error \
--header "Authorization: Bearer $GH_TOKEN" \
--header 'Accept: application/vnd.github+json' \
"https://api.github.com/$1"
}
run=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
[[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]]
[[ $(jq -r '.head_branch' <<<"$run") == main ]]
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(github_api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
[.jobs[] | select(.name == $name)] as $matches
| (($matches | length) == 1 and $matches[0].conclusion == "success")
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
artifact=$(github_api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
[[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]]
[[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]]
[[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]]
@@ -122,31 +141,39 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
run-id: ${{ inputs.build_run_id }}
github-token: ${{ github.token }}
- name: Extract exact RustFS binary
- name: Extract exact RustFS binary and reviewed catalog
shell: bash
env:
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CATALOG_SHA256: ${{ inputs.catalog_sha256 }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ "$CATALOG_SHA256" =~ ^[0-9a-f]{64}$ ]]
short_sha=${SOURCE_SHA:0:7}
package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit)
[[ -n "$package" ]]
mkdir -p binary
unzip -qq "$package" rustfs -d binary
unzip -qq "$package" rustfs rustfs.cpu-symbol-catalog.json rustfs.cpu-symbol-catalog.sha256 -d binary
chmod +x binary/rustfs
printf '%s %s\n' "$BINARY_SHA256" binary/rustfs | sha256sum --check --strict
# The packaged digest is corroboration, never the independent trust input.
[[ $(cat binary/rustfs.cpu-symbol-catalog.sha256) == "$CATALOG_SHA256" ]]
printf '%s %s\n' "$CATALOG_SHA256" binary/rustfs.cpu-symbol-catalog.json | sha256sum --check --strict
chmod 0444 binary/rustfs.cpu-symbol-catalog.json
- name: Run signed service job acceptance
shell: bash
env:
CATALOG_SHA256: ${{ inputs.catalog_sha256 }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
@@ -154,8 +181,10 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_CPU_CATALOG="$GITHUB_WORKSPACE/binary/rustfs.cpu-symbol-catalog.json" \
CONNECT_E2E_CPU_CATALOG_SHA256="$CATALOG_SHA256" \
CONNECT_E2E_PROFILE_EVIDENCE="$GITHUB_WORKSPACE/profile-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=profile
@@ -183,7 +212,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('profile-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-profile-cpu-evidence-${{ github.run_id }}
path: |
@@ -0,0 +1,243 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Connect service memory profile acceptance
on:
workflow_dispatch:
inputs:
build_run_id:
description: Successful main-branch Build and Release run ID
required: true
type: string
artifact_id:
description: Linux x86_64 GNU artifact from that run
required: true
type: string
source_sha:
description: Exact RustFS source commit
required: true
type: string
artifact_digest:
description: GitHub artifact digest including sha256 prefix
required: true
type: string
binary_sha256:
description: Independently verified rustfs binary SHA-256
required: true
type: string
connect_sha:
description: Exact Connect memory acceptance harness commit
required: true
type: string
permissions:
actions: read
contents: read
jobs:
profile-memory:
name: Verify signed service memory profile over mTLS
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
repository: rustfs/connect
path: connect-harness
persist-credentials: false
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Verify official source and artifact identity
shell: bash
env:
GH_TOKEN: ${{ github.token }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
SOURCE_SHA: ${{ inputs.source_sha }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
run: |
set -euo pipefail
[[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
[[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
jq -e --arg source "$SOURCE_SHA" '
.head_sha == $source and .head_branch == "main"
and .head_repository.full_name == "rustfs/rustfs"
and .name == "Build and Release" and .path == ".github/workflows/build.yml"
and .status == "completed" and .conclusion == "success"
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
.workflow_run.id == $run and .workflow_run.head_sha == $source
and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
and .digest == $digest and .expired == false
and (.expires_at | fromdateiso8601) > now
and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
' <<<"$artifact" >/dev/null
jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
[[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
printf '%s official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
- name: Extract only the verified official service binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
run: |
set -euo pipefail
python3 - <<'PY'
import io
import os
import pathlib
import shutil
import stat
import zipfile
def regular(entry):
mode = entry.external_attr >> 16
return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
with zipfile.ZipFile('official-artifact.zip') as outer:
entries = outer.infolist()
assert 1 <= len(entries) <= 16
assert len({entry.filename for entry in entries}) == len(entries)
assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
assert sum(entry.file_size for entry in entries) <= 2147483648
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
shutil.copyfileobj(source, destination)
PY
chmod 0755 binary/rustfs
printf '%s binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
file binary/rustfs | grep -Eq 'ELF 64-bit LSB.*x86-64'
- name: Run the real service job under S3 workload
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
git -C connect-harness diff --exit-code
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
changed=$(git -C connect-harness diff --name-only)
[[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
(cd connect-harness && ./tests/connectivity/profile-memory-evidence.test.sh)
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PROFILE_MEMORY_EVIDENCE="$GITHUB_WORKSPACE/profile-memory-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=profile-memory
- name: Bind and validate sanitized evidence
shell: bash
env:
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
jq -e --arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" '
.provenance.connectSha == $connect and .provenance.sourceSha == $source
and .provenance.binarySha256 == $binary
' profile-memory-evidence.json >/dev/null
jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
--arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
--arg expiry "$(cat artifact-expires-at)" '
.provenance += {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry}
' profile-memory-evidence.json > profile-memory-evidence.bound.json
mv profile-memory-evidence.bound.json profile-memory-evidence.json
node connect-harness/tests/connectivity/profile-memory-evidence.mjs profile-memory-evidence.json --bound
- name: Upload only verified sanitized evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
with:
name: connect-profile-memory-evidence-${{ github.run_id }}
path: profile-memory-evidence.json
retention-days: 14
if-no-files-found: error
@@ -17,6 +17,14 @@ name: Connect profile.threads service-job acceptance
on:
workflow_dispatch:
inputs:
thread_scope:
description: Thread profile scope
required: true
type: choice
default: native
options:
- native
- runtime
build_run_id:
description: Main-branch Build and Release workflow run ID
required: true
@@ -48,24 +56,24 @@ permissions:
jobs:
profile-threads:
name: Verify native Linux x86_64 profile.threads service job
runs-on: ubuntu-latest
name: Verify Linux x86_64 profile.threads service job
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -73,13 +81,54 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Verify source run and artifact identity
shell: bash
env:
@@ -107,7 +156,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +171,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -148,13 +197,19 @@ jobs:
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
THREAD_SCOPE: ${{ inputs.thread_scope }}
run: |
set -euo pipefail
case "$THREAD_SCOPE" in
native) export CONNECT_E2E_THREAD_SCOPE=NATIVE_THREADS ;;
runtime) export CONNECT_E2E_THREAD_SCOPE=TOKIO_RUNTIME ;;
*) exit 1 ;;
esac
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_PROFILE_THREADS_EVIDENCE="$GITHUB_WORKSPACE/profile-threads-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=profile-threads
@@ -168,6 +223,7 @@ jobs:
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
THREAD_SCOPE: ${{ inputs.thread_scope }}
run: |
set -euo pipefail
jq \
@@ -177,13 +233,14 @@ jobs:
--arg artifactDigest "$ARTIFACT_DIGEST" \
--arg connectSha "$CONNECT_SHA" \
--arg sourceSha "$SOURCE_SHA" \
'. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha}' \
--arg requestedThreadScope "$THREAD_SCOPE" \
'. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest, connectSha: $connectSha, sourceSha: $sourceSha, requestedThreadScope: $requestedThreadScope}' \
profile-threads-service-job-evidence.json >profile-threads-service-job-evidence.bound.json
mv profile-threads-service-job-evidence.bound.json profile-threads-service-job-evidence.json
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('profile-threads-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-profile-threads-evidence-${{ github.run_id }}
path: |
@@ -49,23 +49,23 @@ permissions:
jobs:
top-api:
name: Verify native Linux x86_64 top.api service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -73,10 +73,18 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +115,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +130,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -154,7 +162,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_API_EVIDENCE="$GITHUB_WORKSPACE/top-api-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-api
@@ -183,7 +191,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('top-api-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-top-api-evidence-${{ github.run_id }}
path: |
+185 -68
View File
@@ -12,21 +12,21 @@
# See the License for the specific language governing permissions and
# limitations under the License.
name: Connect top.disk artifact acceptance
name: Connect top.disk service-job acceptance
on:
workflow_dispatch:
inputs:
build_run_id:
description: Successful main-branch Build and Release workflow run ID
description: Successful main-branch Build and Release run ID
required: true
type: string
artifact_id:
description: Linux x86_64 GNU artifact ID from that run
description: Linux x86_64 GNU artifact from that run
required: true
type: string
source_sha:
description: Exact 40-character source commit
description: Exact RustFS source commit
required: true
type: string
artifact_digest:
@@ -34,7 +34,11 @@ on:
required: true
type: string
binary_sha256:
description: Expected rustfs binary SHA-256
description: Independently verified rustfs binary SHA-256
required: true
type: string
connect_sha:
description: Exact Connect top.disk service-job acceptance harness commit
required: true
type: string
@@ -44,16 +48,75 @@ permissions:
jobs:
top-disk:
name: Verify native Linux x86_64 top.disk artifact
runs-on: ubuntu-latest
timeout-minutes: 15
name: Verify native top.disk service job over mTLS
runs-on: dind-sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Checkout exact RustFS source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Verify source run and artifact identity
- name: Checkout exact Connect harness
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
persist-credentials: false
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Verify official source and artifact identity
shell: bash
env:
GH_TOKEN: ${{ github.token }}
@@ -61,82 +124,136 @@ jobs:
ARTIFACT_ID: ${{ inputs.artifact_id }}
SOURCE_SHA: ${{ inputs.source_sha }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
run: |
set -euo pipefail
[[ "$BUILD_RUN_ID" =~ ^[0-9]+$ ]]
[[ "$ARTIFACT_ID" =~ ^[0-9]+$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$GITHUB_REPOSITORY" == rustfs/rustfs ]]
[[ "$BUILD_RUN_ID" =~ ^[1-9][0-9]*$ && "$ARTIFACT_ID" =~ ^[1-9][0-9]*$ ]]
[[ "$SOURCE_SHA" =~ ^[0-9a-f]{40}$ && "$CONNECT_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$ARTIFACT_DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
[[ "$BINARY_SHA256" =~ ^[0-9a-f]{64}$ ]]
[[ $(git -C rustfs-source rev-parse HEAD) == "$SOURCE_SHA" ]]
[[ $(git -C connect-harness rev-parse HEAD) == "$CONNECT_SHA" ]]
[[ $(git -C rustfs-source remote get-url origin) == https://github.com/rustfs/rustfs ]]
[[ $(git -C connect-harness remote get-url origin) == https://github.com/rustfs/connect ]]
run=$(gh api "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}")
jq -e --arg source "$SOURCE_SHA" '
.head_sha == $source and .head_branch == "main"
and .head_repository.full_name == "rustfs/rustfs"
and .name == "Build and Release" and .path == ".github/workflows/build.yml"
and .status == "completed" and .conclusion == "success"
' <<<"$run" >/dev/null
jobs=$(gh api --paginate --slurp "repos/rustfs/rustfs/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e '
[.[].jobs[] | select(.name | test("^Build RustFS \\(linux-x86_64-gnu\\)$"))] as $matches
| ($matches | length) == 1 and $matches[0].conclusion == "success"
' <<<"$jobs" >/dev/null
artifact=$(gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}")
jq -e --argjson run "$BUILD_RUN_ID" --arg source "$SOURCE_SHA" --arg digest "$ARTIFACT_DIGEST" '
.workflow_run.id == $run and .workflow_run.head_sha == $source
and .name == ("rustfs-linux-x86_64-gnu-dev-" + $source[0:7])
and .digest == $digest and .expired == false
and (.expires_at | fromdateiso8601) > now
and .size_in_bytes > 0 and .size_in_bytes <= 2147483648
' <<<"$artifact" >/dev/null
jq -r '.expires_at' <<<"$artifact" > artifact-expires-at
gh api "repos/rustfs/rustfs/actions/artifacts/${ARTIFACT_ID}/zip" > official-artifact.zip
[[ $(stat --format=%s official-artifact.zip) == $(jq -r '.size_in_bytes' <<<"$artifact") ]]
printf '%s official-artifact.zip\n' "${ARTIFACT_DIGEST#sha256:}" | sha256sum --check --strict
run=$(gh api "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}")
[[ $(jq -r '.conclusion' <<<"$run") == success ]]
[[ $(jq -r '.head_sha' <<<"$run") == "$SOURCE_SHA" ]]
[[ $(jq -r '.head_branch' <<<"$run") == main ]]
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
artifact=$(gh api "repos/${GITHUB_REPOSITORY}/actions/artifacts/${ARTIFACT_ID}")
[[ $(jq -r '.workflow_run.id' <<<"$artifact") == "$BUILD_RUN_ID" ]]
[[ $(jq -r '.workflow_run.head_sha' <<<"$artifact") == "$SOURCE_SHA" ]]
[[ $(jq -r '.name' <<<"$artifact") == rustfs-linux-x86_64-gnu-* ]]
[[ $(jq -r '.digest' <<<"$artifact") == "$ARTIFACT_DIGEST" ]]
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
run-id: ${{ inputs.build_run_id }}
github-token: ${{ github.token }}
- name: Extract exact RustFS binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
short_sha=${SOURCE_SHA:0:7}
package=$(find artifact -type f -name "rustfs-linux-x86_64-gnu-dev-${short_sha}.zip" -print -quit)
[[ -n "$package" ]]
mkdir -p binary
unzip -qq "$package" rustfs -d binary
chmod +x binary/rustfs
- name: Run top.disk acceptance
- name: Extract only the verified official service binary
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
run: |
set -euo pipefail
scripts/ci/check_connect_top_disk_artifact.sh \
binary/rustfs "$SOURCE_SHA" "$BINARY_SHA256" \
top-disk-runtime-evidence.json top-disk.zip
python3 - <<'PY'
import io
import os
import pathlib
import shutil
import stat
import zipfile
- name: Bind workflow and artifact provenance
def regular(entry):
mode = entry.external_attr >> 16
return not entry.is_dir() and stat.S_IFMT(mode) in (0, stat.S_IFREG)
package = f"rustfs-linux-x86_64-gnu-dev-{os.environ['SOURCE_SHA'][:7]}.zip"
with zipfile.ZipFile('official-artifact.zip') as outer:
entries = outer.infolist()
assert 1 <= len(entries) <= 16
assert len({entry.filename for entry in entries}) == len(entries)
assert all(regular(entry) and pathlib.PurePosixPath(entry.filename).name == entry.filename for entry in entries)
assert sum(entry.file_size for entry in entries) <= 2147483648
assert outer.testzip() is None
with zipfile.ZipFile(io.BytesIO(outer.read(package))) as inner:
entries = inner.infolist()
expected = {'rustfs', 'rustfs.cpu-symbol-catalog.json', 'rustfs.cpu-symbol-catalog.sha256'}
assert {entry.filename for entry in entries} == expected and len(entries) == len(expected)
assert all(regular(entry) and 0 < entry.file_size <= 1073741824 for entry in entries)
assert inner.testzip() is None
pathlib.Path('binary').mkdir()
with inner.open('rustfs') as source, open('binary/rustfs', 'xb') as destination:
header = source.read(20)
assert header[:7] == b'\x7fELF\x02\x01\x01' and header[18:20] == b'\x3e\x00', 'Expected a little-endian ELF64 x86-64 binary'
destination.write(header)
shutil.copyfileobj(source, destination)
PY
chmod 0755 binary/rustfs
printf '%s binary/rustfs\n' "$BINARY_SHA256" | sha256sum --check --strict
- name: Run the real top.disk service job
shell: bash
env:
SOURCE_SHA: ${{ inputs.source_sha }}
CONNECT_E2E_PRODUCER_SOURCE: ${{ inputs.source_sha }}
CONNECT_E2E_PRODUCER_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_E2E_PRODUCER_ARTIFACT_ID: ${{ inputs.artifact_id }}
run: |
set -euo pipefail
git -C connect-harness diff --exit-code
printf '%s\n' "$SOURCE_SHA" >connect-harness/tests/e2e/connected/rustfs-ref
changed=$(git -C connect-harness diff --name-only)
[[ -z "$changed" || "$changed" == tests/e2e/connected/rustfs-ref ]]
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_DISK_EVIDENCE="$GITHUB_WORKSPACE/top-disk-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-disk
- name: Bind and validate sanitized evidence
shell: bash
env:
BUILD_RUN_ID: ${{ inputs.build_run_id }}
ARTIFACT_ID: ${{ inputs.artifact_id }}
ARTIFACT_DIGEST: ${{ inputs.artifact_digest }}
BINARY_SHA256: ${{ inputs.binary_sha256 }}
CONNECT_SHA: ${{ inputs.connect_sha }}
SOURCE_SHA: ${{ inputs.source_sha }}
run: |
set -euo pipefail
jq \
--arg workflowRunId "$GITHUB_RUN_ID" \
--arg buildRunId "$BUILD_RUN_ID" \
--arg artifactId "$ARTIFACT_ID" \
--arg artifactDigest "$ARTIFACT_DIGEST" \
'. + {workflowRunId: $workflowRunId, buildRunId: $buildRunId, artifactId: $artifactId, artifactDigest: $artifactDigest}' \
top-disk-runtime-evidence.json >top-disk-runtime-evidence.bound.json
mv top-disk-runtime-evidence.bound.json top-disk-runtime-evidence.json
jq -e --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" --arg artifact "$ARTIFACT_ID" '
.status == "PASS" and .officialArtifactId == $artifact and .producerSource == $source and .producerSha256 == $binary
and .result.provenance.sourceCommit == $source
and .result.provenance.executableSha256 == $binary
' top-disk-service-job-evidence.json >/dev/null
jq --arg run "$GITHUB_RUN_ID" --arg build "$BUILD_RUN_ID" \
--arg artifact "$ARTIFACT_ID" --arg digest "$ARTIFACT_DIGEST" \
--arg connect "$CONNECT_SHA" --arg source "$SOURCE_SHA" --arg binary "$BINARY_SHA256" \
--arg expiry "$(cat artifact-expires-at)" '
. + {workflowRunId:$run,buildRunId:$build,artifactId:$artifact,artifactDigest:$digest,artifactExpiresAt:$expiry,connectSha:$connect,sourceSha:$source,binarySha256:$binary}
' top-disk-service-job-evidence.json > top-disk-service-job-evidence.bound.json
mv top-disk-service-job-evidence.bound.json top-disk-service-job-evidence.json
- name: Upload acceptance evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
- name: Upload only verified sanitized evidence
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-top-disk-evidence-${{ github.run_id }}
path: |
top-disk-runtime-evidence.json
top-disk.zip
name: connect-top-disk-service-job-evidence-${{ github.run_id }}
path: top-disk-service-job-evidence.json
retention-days: 14
if-no-files-found: error
@@ -49,23 +49,23 @@ permissions:
jobs:
top-locks:
name: Verify native Linux x86_64 top.locks service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -73,10 +73,18 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +115,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +130,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -154,7 +162,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_LOCKS_EVIDENCE="$GITHUB_WORKSPACE/top-locks-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-locks
@@ -183,7 +191,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('top-locks-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-top-locks-evidence-${{ github.run_id }}
path: |
@@ -49,23 +49,23 @@ permissions:
jobs:
top-rpc:
name: Verify native Linux x86_64 top.rpc service job
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 45
steps:
- name: Checkout acceptance harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Checkout exact RustFS source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
path: rustfs-source
persist-credentials: false
ref: ${{ inputs.source_sha }}
- name: Checkout exact Connect harness
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/connect
path: connect-harness
@@ -73,10 +73,18 @@ jobs:
ref: ${{ inputs.connect_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
- name: Set up Docker Compose
uses: ./.github/actions/setup-docker-compose
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 22.22.2
node-version: 25.8.1
cache: npm
cache-dependency-path: connect-harness/web/package-lock.json
@@ -107,7 +115,7 @@ jobs:
[[ $(jq -r '.head_repository.full_name' <<<"$run") == "$GITHUB_REPOSITORY" ]]
[[ $(jq -r '.name' <<<"$run") == "Build and Release" ]]
expected_job='Build RustFS (linux-x86_64-gnu, sm-standard-2, x86_64-unknown-linux-gnu, false, linux, pyroscope)'
expected_job='Build RustFS (linux-x86_64-gnu)'
jobs=$(gh api --paginate --slurp "repos/${GITHUB_REPOSITORY}/actions/runs/${BUILD_RUN_ID}/jobs?per_page=100")
jq -e --arg name "$expected_job" '
[.[].jobs[] | select(.name == $name)] as $matches
@@ -122,7 +130,7 @@ jobs:
[[ $(jq -r '.expired' <<<"$artifact") == false ]]
- name: Download exact build artifact
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
artifact-ids: ${{ inputs.artifact_id }}
path: artifact
@@ -154,7 +162,7 @@ jobs:
npm --prefix connect-harness/web ci
connect-harness/web/node_modules/.bin/playwright install --with-deps chromium
make -C connect-harness e2e-connected-dispatch-check
RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
TMPDIR="$GITHUB_WORKSPACE" RUSTFS_BINARY="$GITHUB_WORKSPACE/binary/rustfs" \
RUSTFS_WORKTREE="$GITHUB_WORKSPACE/rustfs-source" \
CONNECT_E2E_TOP_RPC_EVIDENCE="$GITHUB_WORKSPACE/top-rpc-service-job-evidence.json" \
make -C connect-harness e2e-connected E2E_SCENARIO=top-rpc
@@ -183,7 +191,7 @@ jobs:
- name: Upload acceptance evidence
if: ${{ always() && hashFiles('top-rpc-service-job-evidence.json') != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: connect-top-rpc-evidence-${{ github.run_id }}
path: |
+4 -4
View File
@@ -76,7 +76,7 @@ jobs:
NEXTEST_PROFILE: ci
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -89,7 +89,7 @@ jobs:
install-build-packaging-tools: 'false'
- name: Install cargo-llvm-cov
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
with:
tool: cargo-llvm-cov
@@ -112,7 +112,7 @@ jobs:
- name: Upload coverage artifact
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: coverage-lcov-${{ github.run_number }}
path: |
@@ -134,7 +134,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+3 -51
View File
@@ -96,7 +96,7 @@ jobs:
source_ref: ${{ steps.check.outputs.source_ref }}
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
# For workflow_run events, checkout the specific commit that triggered the workflow
@@ -352,7 +352,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
@@ -544,61 +544,13 @@ jobs:
category: container-image-${{ matrix.variant }}
- name: Upload container scan report
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: container-image-scan-${{ matrix.variant }}
path: trivy-${{ matrix.variant }}.sarif
if-no-files-found: error
retention-days: 30
# Docker Hub's overview is a separate `full_description` field that
# `docker push` never touches, so it drifted from README.md and kept
# linking to pages that no longer exist (#7995). Republish it from the
# commit the images were built from, only when images were pushed.
sync-dockerhub-description:
name: Sync Docker Hub Description
needs: [ build-check, build-docker ]
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
sparse-checkout: README.md
sparse-checkout-cone-mode: false
- name: Sync Docker Hub description
uses: peter-evans/dockerhub-description@1b9a80c056b620d92cedb9d9b5a223409c68ddfa # v5.0.0
with:
username: ${{ env.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
repository: ${{ env.REGISTRY_DOCKERHUB }}
readme-filepath: ./README.md
# README.md links to docs/ and CONTRIBUTING.md by relative path;
# rewrite those to github.com URLs so they resolve on Docker Hub.
enable-url-completion: true
# Docker Hub caps `full_description` at 25,000 bytes and the action
# truncates to fit with only a warning, which would publish a README
# cut off mid-sentence. README.md is already near the cap, so read the
# published overview back and fail if it hit the limit.
- name: Check published description was not truncated
env:
REPOSITORY: ${{ env.REGISTRY_DOCKERHUB }}
run: |
limit=25000
published=$(curl -sSf "https://hub.docker.com/v2/repositories/${REPOSITORY}/" \
| jq -j '.full_description' | wc -c | tr -d ' ')
echo "Published overview: ${published} bytes (Docker Hub limit: ${limit})"
# Truncation lands within one UTF-8 character of the cap.
if [ "$published" -ge $((limit - 4)) ]; then
echo "::error file=README.md::Docker Hub overview was truncated at ${limit} bytes; trim README.md before the next release."
exit 1
fi
# Docker build summary
docker-summary:
name: Docker Build Summary
+11 -7
View File
@@ -38,6 +38,7 @@ on:
- "Cargo.lock"
- "Cargo.toml"
- ".config/nextest.toml"
- ".config/e2e-distributed-selection.txt"
- ".github/workflows/e2e-distributed.yml"
- "crates/audit/**"
- "crates/common/**"
@@ -77,6 +78,7 @@ concurrency:
jobs:
distributed:
if: github.repository == 'rustfs/rustfs'
name: Distributed 4-node 4-disk e2e
# GitHub-hosted VM: loop and tmpfs mounts work here. sm-standard-4 is an
# ARC pod and rejects both (`mount -o loop` ENOENT, tmpfs "read-only").
@@ -93,7 +95,7 @@ jobs:
UPGRADE_SOURCE_SHA256: 7c789386bf85278f865b8e0d359bf4edb84d5aa408cc3fa54a18c25ca74cd6e7
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -151,7 +153,7 @@ jobs:
- name: Build rustfs binary
run: |
python3 scripts/e2e_binary.py build --bins
python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
- name: Verify distributed e2e membership
env:
@@ -166,15 +168,17 @@ jobs:
FILTER: ${{ inputs.filter }}
run: |
set -euo pipefail
rm -f target/nextest/e2e-distributed/junit.xml
if [ -n "${FILTER}" ]; then
python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test -E "${FILTER}"
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-distributed -p e2e_test -E "${FILTER}"
else
python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-distributed -p e2e_test --no-tests=fail
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-distributed -p e2e_test --no-tests=fail
fi
test -s target/nextest/e2e-distributed/junit.xml
- name: Upload distributed e2e diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-distributed-${{ github.run_number }}
path: |
@@ -199,14 +203,14 @@ jobs:
alert-on-failure:
name: Alert on scheduled failure
needs: [distributed]
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+12 -9
View File
@@ -53,6 +53,7 @@ concurrency:
jobs:
repl-nightly:
if: github.repository == 'rustfs/rustfs'
name: Replication e2e (nightly)
# dual-node tests spawn two full rustfs servers each; the extra headroom
# over the per-PR sm-standard-2 e2e job keeps the parallel process fan-out
@@ -63,7 +64,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -77,7 +78,7 @@ jobs:
# The STS dual-node test requires awscurl and fails if it is unavailable.
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -108,7 +109,7 @@ jobs:
- name: Upload nextest junit report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-replication-nightly-junit-${{ github.run_number }}
path: |
@@ -119,6 +120,7 @@ jobs:
if-no-files-found: ignore
cluster-nightly:
if: github.repository == 'rustfs/rustfs'
name: Cluster fault e2e (nightly)
runs-on: sm-standard-4
timeout-minutes: 90
@@ -126,7 +128,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -157,7 +159,7 @@ jobs:
- name: Upload cluster fault diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-cluster-nightly-${{ github.run_number }}
path: |
@@ -168,6 +170,7 @@ jobs:
if-no-files-found: warn
protocols-nightly:
if: github.repository == 'rustfs/rustfs'
name: Protocol e2e (nightly)
runs-on: sm-standard-4
timeout-minutes: 90
@@ -176,7 +179,7 @@ jobs:
RUSTFS_BUILD_FEATURES: ftps,webdav,sftp
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -213,7 +216,7 @@ jobs:
- name: Upload protocol diagnostics
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-protocol-nightly-${{ github.run_number }}
path: |
@@ -229,14 +232,14 @@ jobs:
# Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
# manual workflow_dispatch runs stay quiet so a debugging run never files a
# spurious alert.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+13 -8
View File
@@ -159,7 +159,7 @@ jobs:
TEST_MODE: ${{ matrix.test-mode }}
S3_SHARD_INDEX: ${{ matrix.shard-index }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -167,7 +167,7 @@ jobs:
# ship a working pip (ci-1: a bare python3 without pip is what broke the
# scheduled sweep). Keeps the workflow correct across runner drift.
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -264,7 +264,7 @@ jobs:
hostname: rustfs4
volumes: [rustfs4-data:/data]
lb:
image: haproxy:2.9
image: haproxy:3.4.4
hostname: lb
networks: [rustfs-net]
ports:
@@ -308,6 +308,11 @@ jobs:
docker compose -f compose.yml up -d
# Validate the pinned HAProxy image parses the exact config before
# the S3 compatibility sweep starts.
docker compose -f compose.yml exec -T lb \
haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg
- name: Wait for RustFS ready
run: |
for _ in {1..120}; do
@@ -357,7 +362,7 @@ jobs:
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: s3tests-${{ env.TEST_MODE }}-shard-${{ matrix.shard-index }}
path: artifacts/**
@@ -369,12 +374,12 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
@@ -435,7 +440,7 @@ jobs:
- name: Upload canary artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: s3tests-upstream-head
path: artifacts/s3tests-upstream-head/**
@@ -454,7 +459,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+111 -61
View File
@@ -25,6 +25,7 @@ on:
- "crates/e2e_test/src/upgrade_compatibility_test.rs"
- "crates/ecstore/**"
- "crates/filemeta/**"
- "crates/heal/**"
- "crates/kms/**"
- "crates/storage-api/**"
- "rustfs/**"
@@ -46,76 +47,95 @@ concurrency:
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
UPGRADE_SOURCE_VERSION: 1.0.0-rc.5
UPGRADE_SOURCE_ASSET: rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip
UPGRADE_SOURCE_SHA256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d
jobs:
upgrade:
name: ${{ matrix.name }}
strategy:
fail-fast: false
matrix:
include:
# The two `_from_rc2_` tests keep their names: they assert
# release-independent object contracts and pass unchanged against the
# newer pinned source, so renaming them would only churn history and
# the CI required-check names. UPGRADE_SOURCE_VERSION above is the
# single source of truth for which release they actually run against.
- name: Direct upgrade from the previous release
cache_key: e2e-direct-upgrade
test: direct_upgrade_from_rc2_preserves_object_contracts
artifact: direct-upgrade
- name: Mixed-version rolling upgrade from the previous release
cache_key: e2e-mixed-version-upgrade
test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
artifact: mixed-version-upgrade
- name: Bucket configuration survives the upgrade
cache_key: e2e-bucket-config-upgrade
test: direct_upgrade_from_previous_release_preserves_bucket_configuration
artifact: bucket-config-upgrade
- name: Rollback reads current bucket metadata
cache_key: e2e-bucket-config-rollback
test: rollback_to_previous_release_reads_current_bucket_metadata
artifact: bucket-config-rollback
- name: ODM configuration recovery after rc.5 rollback
cache_key: e2e-odm-config-rollback
test: rc5_rollback_requires_restoring_odm_configuration
artifact: odm-config-rollback
- name: Multipart layouts survive the rc.5 upgrade
cache_key: e2e-multipart-layout-upgrade
test: direct_upgrade_from_rc5_preserves_multipart_layouts
artifact: multipart-layout-upgrade
- name: rc.5 multipart replication baseline
cache_key: e2e-multipart-layout-baseline
test: rc5_baseline_replicates_multipart_layouts
artifact: multipart-layout-baseline
build:
name: Build upgrade candidate
runs-on: ubuntu-latest
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-shared-key: ${{ matrix.cache_key }}
cache-shared-key: e2e-upgrade-server-hosted
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Download pinned previous release
- name: Build current RustFS binary
run: python3 scripts/e2e_binary.py build
- name: Upload verified upgrade candidate
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: upgrade-candidate-${{ github.run_id }}
path: |
target/debug/rustfs
target/debug/rustfs.e2e.json
if-no-files-found: error
retention-days: 3
compression-level: 0
overwrite: true
upgrade:
name: Upgrade compatibility (${{ matrix.source.version }})
needs: build
runs-on: ubuntu-latest
timeout-minutes: 60
strategy:
fail-fast: false
matrix:
source:
- version: 1.0.0-rc.5
asset: rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip
sha256: 3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d
- version: 1.0.1
asset: rustfs-linux-x86_64-gnu-v1.0.1.zip
sha256: 9742e183cb0dfb0d48fa909797af572fe47b7f827569036baa4156e0d3958ee5
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
UPGRADE_SOURCE_VERSION: ${{ matrix.source.version }}
UPGRADE_SOURCE_ASSET: ${{ matrix.source.asset }}
UPGRADE_SOURCE_SHA256: ${{ matrix.source.sha256 }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-shared-key: e2e-upgrade-tests-hosted
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
- name: Download verified upgrade candidate
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: upgrade-candidate-${{ github.run_id }}
path: target/debug
- name: Restore candidate executable bit
run: chmod +x target/debug/rustfs
- name: Verify and unpack pinned previous release
env:
SOURCE_DIR: ${{ runner.temp }}/rustfs-upgrade-source
run: |
set -euo pipefail
mkdir -p "$SOURCE_DIR"
archive="$SOURCE_DIR/$UPGRADE_SOURCE_ASSET"
archive="target/debug/$UPGRADE_SOURCE_ASSET"
curl --fail --location --retry 3 --output "$archive" \
"https://github.com/${GITHUB_REPOSITORY}/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
"https://github.com/rustfs/rustfs/releases/download/${UPGRADE_SOURCE_VERSION}/${UPGRADE_SOURCE_ASSET}"
echo "$UPGRADE_SOURCE_SHA256 $archive" | sha256sum --check --strict
unzip -q "$archive" -d "$SOURCE_DIR"
chmod +x "$SOURCE_DIR/rustfs"
@@ -123,32 +143,62 @@ jobs:
echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
- name: Build current RustFS binary
run: |
python3 scripts/e2e_binary.py build
- name: Run upgrade compatibility test
# Reuse one compiled harness, while keeping each case in its own process.
# Continue after a failure to retain the former matrix's complete verdict.
- name: Run upgrade compatibility tests
shell: bash
env:
RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
NO_PROXY: 127.0.0.1,localhost,::1
run: |
python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
"upgrade_compatibility_test::${{ matrix.test }}" \
-- --ignored --exact --nocapture
export no_proxy="$NO_PROXY"
cases=(
direct-upgrade:direct_upgrade_from_previous_release_preserves_object_contracts
mixed-version-upgrade:rolling_upgrade_from_previous_release_preserves_mixed_version_contracts
bucket-config-upgrade:direct_upgrade_from_previous_release_preserves_bucket_configuration
bucket-config-rollback:rollback_to_previous_release_reads_current_bucket_metadata
multipart-layout-upgrade:direct_upgrade_from_previous_release_preserves_multipart_layouts
multipart-layout-baseline:previous_release_baseline_replicates_multipart_layouts
)
if [[ "$UPGRADE_SOURCE_VERSION" == "1.0.0-rc.5" ]]; then
cases+=(odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration)
cases+=(checkpoint-upgrade:rc5_checkpoint_upgrade_rewinds_signed_progress_without_losing_objects)
else
cases+=(legacy-delete-reservation:stable_upgrade_deletes_foreign_reservations_without_losing_live_objects)
fi
status=0
for entry in "${cases[@]}"; do
artifact="${entry%%:*}"
test="${entry#*:}"
export RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR="$RUNNER_TEMP/rustfs-upgrade-g09-evidence/$artifact"
export RUSTFS_E2E_LOG_DIR="$RUNNER_TEMP/rustfs-upgrade-logs/$artifact"
mkdir -p "$RUSTFS_E2E_LOG_DIR"
echo "::group::$test"
if python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
"upgrade_compatibility_test::$test" -- --ignored --exact --nocapture \
2>&1 | tee "$RUSTFS_E2E_LOG_DIR/test.log"; then
echo "PASS: $test"
else
echo "::error::Upgrade compatibility failed: $test"
status=1
fi
echo "::endgroup::"
done
exit "$status"
- name: Upload scanner/heal G09 evidence
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
name: upgrade-g09-evidence-${{ matrix.source.version }}-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence
if-no-files-found: ignore
retention-days: 14
- name: Upload server logs
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }}
name: upgrade-server-logs-${{ matrix.source.version }}-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-logs
if-no-files-found: warn
retention-days: 14
+36 -3
View File
@@ -11,12 +11,45 @@ concurrency:
cancel-in-progress: false
jobs:
collect:
runs-on: ubuntu-latest
if: github.repository == 'rustfs/rustfs'
runs-on: smoke-testing
timeout-minutes: 15
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Ensure GitHub CLI
shell: bash
run: |
if command -v gh >/dev/null 2>&1; then
gh --version
exit 0
fi
gh_version="2.101.0"
case "$(uname -m)" in
x86_64) gh_arch="amd64" ;;
aarch64|arm64) gh_arch="arm64" ;;
*)
echo "Unsupported Linux architecture for GitHub CLI: $(uname -m)" >&2
exit 1
;;
esac
gh_install_root="${RUNNER_TEMP:-$PWD}/gh-cli"
rm -rf "$gh_install_root"
mkdir -p "$gh_install_root/bin"
archive="$gh_install_root/gh.tar.gz"
curl --fail --location --retry 3 \
"https://github.com/cli/cli/releases/download/v${gh_version}/gh_${gh_version}_linux_${gh_arch}.tar.gz" \
--output "$archive"
tar -xzf "$archive" -C "$gh_install_root"
install -m 0755 \
"$gh_install_root/gh_${gh_version}_linux_${gh_arch}/bin/gh" \
"$gh_install_root/bin/gh"
echo "$gh_install_root/bin" >> "$GITHUB_PATH"
export PATH="$gh_install_root/bin:$PATH"
gh --version
- name: Collect and publish verified chain health
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
@@ -24,7 +57,7 @@ jobs:
run: python3 scripts/functional_chain_health.py --source-ref "${NIGHTLY_SOURCE_REF}" --publish --output "${RUNNER_TEMP}/chain-health.json"
- name: Retain health observation
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-health-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-health.json
+16 -14
View File
@@ -72,16 +72,18 @@ jobs:
fuzz-build:
name: Build Fuzz Harness
if: >
(github.event_name == 'pull_request' && github.event.action != 'closed') ||
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch'
github.repository == 'rustfs/rustfs' && (
(github.event_name == 'pull_request' && github.event.action != 'closed') ||
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch'
)
runs-on: sm-standard-4
timeout-minutes: 60
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -93,7 +95,7 @@ jobs:
cache-save-if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }}
- name: Install cargo-fuzz
uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
uses: taiki-e/install-action@7623a79cdfecb99d681017af368ca353d9f49bb5 # v2
with:
tool: cargo-fuzz
@@ -113,7 +115,7 @@ jobs:
done
- name: Upload prebuilt fuzz binaries
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: fuzz-prebuilt-binaries-${{ github.run_number }}
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/
@@ -143,12 +145,12 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: fuzz-prebuilt-binaries-${{ github.run_number }}
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
@@ -167,7 +169,7 @@ jobs:
- name: Upload fuzz smoke artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: fuzz-smoke-${{ matrix.target }}-${{ github.run_number }}
path: |
@@ -197,12 +199,12 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download prebuilt fuzz binaries
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: fuzz-prebuilt-binaries-${{ github.run_number }}
path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
@@ -221,7 +223,7 @@ jobs:
- name: Upload nightly fuzz artifacts
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: fuzz-nightly-${{ matrix.target }}-${{ github.run_number }}
path: |
@@ -239,14 +241,14 @@ jobs:
# `always()` is required: without it this job is skipped when a needed
# job fails. Alerts only for scheduled (nightly) runs (backlog#1149
# ci-8); PR and manual dispatch failures are already watched by a human.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
@@ -0,0 +1,163 @@
# Copyright 2026 RustFS Team
# SPDX-License-Identifier: Apache-2.0
name: Gateway Integration Sync
# Merges `main` into the long-lived `gateway/integration` branch every weekday
# so the s3s -> gateway migration (rustfs/backlog#2734, task T0.1 in #2736)
# never accumulates merge debt. A clean merge lands as a `--no-ff` merge commit
# pushed straight to the branch. A conflicting merge is aborted and fails the
# run after its summary records the source and target SHAs, run URL, and exact
# conflicting paths. It creates no sync branch or pull request. Resolve the
# merge in an isolated worktree based on the latest target, preserve the
# two-parent merge commit, and push only after the required checks pass and the
# target still matches the merge's first parent. If the target moves, restart
# from its new head.
#
# Write scope: only `gateway/integration` is ever pushed. The push uses a
# one-hour installation token of the organisation's GitHub App (contents:
# write) because that app is the only non-admin bypass actor of the
# `gateway/integration` ruleset; GITHUB_TOKEN cannot be a ruleset bypass actor,
# so it cannot push through the required reviews and checks. GITHUB_TOKEN is
# read-only here.
#
# `schedule` fires only from the default branch, so the cron is live once this
# file exists on `main`. `workflow_dispatch` runs from any ref that carries the
# file and always acts on `gateway/integration`.
on:
schedule:
- cron: "0 2 * * 1-5" # Weekdays 02:00 UTC
workflow_dispatch:
inputs:
source_ref:
description: >-
Branch to merge into gateway/integration. The schedule always merges
main; only the conflict acceptance test (rustfs/backlog#2736) sets
another branch.
required: false
default: main
type: string
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: false
env:
TARGET_BRANCH: gateway/integration
jobs:
sync:
name: Merge main into gateway/integration
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Create the branch push token
id: push-token
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3
with:
app-id: ${{ vars.CLA_BOT_APP_ID }}
private-key: ${{ secrets.CLA_BOT_APP_PRIVATE_KEY }}
owner: ${{ github.repository_owner }}
repositories: rustfs
permission-contents: write
- name: Checkout gateway/integration
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push
# credential — the merge commit and the sync branch are pushed with it
# below. It is a one-hour installation token scoped to contents.
ref: gateway/integration
fetch-depth: 0
token: ${{ steps.push-token.outputs.token }}
- name: Configure the bot identity
run: |
git config user.name "rustfs-cla-assistant[bot]"
git config user.email "273121464+rustfs-cla-assistant[bot]@users.noreply.github.com"
- name: Merge the source branch
id: merge
env:
SOURCE_REF: ${{ inputs.source_ref || 'main' }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
set -euo pipefail
if ! [[ "$SOURCE_REF" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] \
|| [[ "$SOURCE_REF" == *..* ]] \
|| [[ "$SOURCE_REF" == "$TARGET_BRANCH" ]]; then
echo "::error::refusing source_ref '${SOURCE_REF}'"
exit 1
fi
git fetch origin "refs/heads/${SOURCE_REF}:refs/remotes/origin/${SOURCE_REF}"
source_sha=$(git rev-parse "origin/${SOURCE_REF}")
target_sha=$(git rev-parse HEAD)
{
echo "source_ref=${SOURCE_REF}"
echo "source_sha=${source_sha}"
echo "target_sha=${target_sha}"
} >> "$GITHUB_OUTPUT"
if git merge --no-ff --no-edit "origin/${SOURCE_REF}"; then
if [[ "$(git rev-parse HEAD)" == "$target_sha" ]]; then
echo "result=up-to-date" >> "$GITHUB_OUTPUT"
echo "\`${TARGET_BRANCH}\` already contains \`${SOURCE_REF}\` (${source_sha}); nothing to push." >> "$GITHUB_STEP_SUMMARY"
else
echo "result=merged" >> "$GITHUB_OUTPUT"
fi
exit 0
fi
conflicts=$(git diff --name-only --diff-filter=U)
if [[ -z "$conflicts" ]]; then
git merge --abort || true
echo "::error::git merge failed without reporting conflicting paths; this is not a conflict"
exit 1
fi
{
echo "## Gateway integration sync stopped on conflicts"
echo
echo "- Run: ${RUN_URL}"
echo "- Source: \`${SOURCE_REF}\` at ${source_sha}"
echo "- Target: \`${TARGET_BRANCH}\` at ${target_sha}"
echo
echo "Conflicting files:"
printf '%s\n' "$conflicts" | sed 's/^/- /'
} >> "$GITHUB_STEP_SUMMARY"
git merge --abort
echo "::error::${SOURCE_REF} cannot be merged cleanly into ${TARGET_BRANCH}; no branch or pull request was created"
exit 1
- name: Push the merge commit
if: steps.merge.outputs.result == 'merged'
env:
SOURCE_REF: ${{ steps.merge.outputs.source_ref }}
SOURCE_SHA: ${{ steps.merge.outputs.source_sha }}
run: |
set -euo pipefail
git push origin HEAD:refs/heads/gateway/integration
echo "Merged \`${SOURCE_REF}\` (${SOURCE_SHA}) into \`${TARGET_BRANCH}\` as $(git rev-parse HEAD)." >> "$GITHUB_STEP_SUMMARY"
alert-on-failure:
name: Alert on scheduled failure
needs: [sync]
# Same ci-8 mechanism audit.yml and coverage.yml use: a scheduled run that
# could not push or encountered merge conflicts must not stay visible only
# in the Actions tab.
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}
+58 -6
View File
@@ -48,9 +48,10 @@ jobs:
chart_version: ${{ steps.version.outputs.chart_version }}
steps:
- name: Checkout helm chart repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- name: Checkout release source
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
ref: ${{ github.event.workflow_run.head_sha || (startsWith(inputs.version, 'refs/tags/') && inputs.version || format('refs/tags/{0}', inputs.version)) }}
persist-credentials: false
# Both inputs reach the shell through env rather than `${{ }}`
@@ -83,6 +84,40 @@ jobs:
./scripts/helm_chart_version.sh "$RAW_TAG"
# Docker and Helm share the binary-build trigger. Do not publish a chart
# until the image selected by its default appVersion is pullable.
- name: Wait for release image
timeout-minutes: 20
env:
APP_VERSION: ${{ steps.version.outputs.app_version }}
run: |
set -euo pipefail
if [[ ! "$APP_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z.-]+)?$ ]] ||
[[ "$APP_VERSION" == *-preview* ]]; then
echo "Invalid deliverable image version: $APP_VERSION" >&2
exit 1
fi
for attempt in {1..60}; do
if token=$(curl --fail --silent --show-error --connect-timeout 10 --max-time 20 \
'https://auth.docker.io/token?service=registry.docker.io&scope=repository:rustfs/rustfs:pull' \
| jq -er '.token // empty') &&
curl --fail --silent --show-error --head --connect-timeout 10 --max-time 20 \
--header "Authorization: Bearer $token" \
--header 'Accept: application/vnd.oci.image.index.v1+json, application/vnd.docker.distribution.manifest.list.v2+json' \
"https://registry-1.docker.io/v2/rustfs/rustfs/manifests/$APP_VERSION" > /dev/null; then
echo "Release image is available: rustfs/rustfs:$APP_VERSION"
exit 0
fi
echo "Waiting for rustfs/rustfs:$APP_VERSION ($attempt/60)"
if (( attempt < 60 )); then
sleep 15
fi
done
echo "Release image is unavailable; refusing to publish Helm chart" >&2
exit 1
- name: Replace chart version and app version
env:
CHART_VERSION: ${{ steps.version.outputs.chart_version }}
@@ -95,6 +130,21 @@ jobs:
- name: Set up Helm
uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
- name: Install yq
run: |
set -euo pipefail
yq_version="4.54.1"
yq_sha256="8e34fc298390875de416e6a4afcb8cabeceb25d9aa8506c1a2f9353cf702ea5f"
tools_dir="$RUNNER_TEMP/rustfs-helm-tools"
mkdir -p "$tools_dir"
curl --fail --silent --show-error --location --retry 3 \
"https://github.com/mikefarah/yq/releases/download/v${yq_version}/yq_linux_amd64" \
--output "$tools_dir/yq"
printf '%s %s\n' "$yq_sha256" "$tools_dir/yq" | sha256sum --check -
chmod +x "$tools_dir/yq"
echo "$tools_dir" >> "$GITHUB_PATH"
"$tools_dir/yq" --version
- name: Test Helm Chart Templates
run: ./scripts/test_helm_templates.sh
@@ -107,10 +157,12 @@ jobs:
--version "${{ steps.version.outputs.chart_version }}"
- name: Upload helm package as artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: helm-package
path: helm/rustfs/*.tgz
path: |
helm/rustfs/*.tgz
helm/rustfs/README.md
retention-days: 1
publish-helm-package:
@@ -121,7 +173,7 @@ jobs:
steps:
- name: Checkout helm package repo
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
# persist-credentials-exempt: this checkout's token IS the push credential —
# the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
@@ -129,7 +181,7 @@ jobs:
token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
- name: Download helm package
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: helm-package
path: ./
+3 -3
View File
@@ -60,7 +60,7 @@ permissions:
jobs:
minio-interop:
name: MinIO interop (EC + SSE read parity)
# Skip on forks: needs the repo's runners and is not a contributor gate.
# Keep this scheduled interoperability lane scoped to the upstream repo.
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 40
@@ -83,7 +83,7 @@ jobs:
INTEROP_REQUIRED_TESTS: '["reads_minio_generated_sse_s3_multipart_fixture", "reads_minio_generated_sse_kms_multipart_fixture", "rejects_minio_generated_sse_s3_fixture_with_wrong_kms_key", "rejects_minio_generated_sse_s3_fixture_with_truncated_ciphertext"]'
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -137,7 +137,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+4 -4
View File
@@ -114,10 +114,10 @@ jobs:
# label was abandoned (no docker.sock on the pod its only run landed on).
# TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
# run) is added by the ci-8 composite action; do not implement it here.
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 120
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -246,7 +246,7 @@ jobs:
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: mint
path: artifacts/**
@@ -264,7 +264,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+32 -15
View File
@@ -43,6 +43,7 @@ env:
jobs:
resolve-source:
if: github.repository == 'rustfs/rustfs'
name: Resolve nightly source
runs-on: ubuntu-latest
timeout-minutes: 10
@@ -51,7 +52,7 @@ jobs:
source_ref: ${{ steps.source.outputs.ref }}
steps:
- name: Checkout selected source
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ env.NIGHTLY_BUILD_REF }}
@@ -71,7 +72,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.resolve-source.outputs.source_sha }}
@@ -79,13 +80,24 @@ jobs:
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
cache-shared-key: build-x86_64-unknown-linux-gnu
cache-shared-key: build-x86_64-unknown-linux-gnu-glibc-2.31
cache-save-if: 'false'
install-cross-tools: 'true'
target: x86_64-unknown-linux-gnu
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Build RustFS
run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
# Pin the glibc baseline (`<target>.<glibc>` for cargo-zigbuild) so
# nightlies run on older distros such as RHEL/Rocky 9 (glibc 2.34); #8062.
run: cargo zigbuild --release --locked --target x86_64-unknown-linux-gnu.2.31 -p rustfs --bins
- name: Verify glibc symbol baseline
run: |
python3 scripts/release/check_glibc.py 2.31 \
target/x86_64-unknown-linux-gnu/release/rustfs \
target/x86_64-unknown-linux-gnu/release/rustfs-cli
target/x86_64-unknown-linux-gnu/release/rustfs --version | grep -Fx 'build target : x86_64-unknown-linux-gnu'
- name: Build DEB package
id: deb
@@ -237,12 +249,12 @@ jobs:
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
- name: Upload DEB artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ steps.deb.outputs.deb_file }}
path: ${{ steps.deb.outputs.deb_file }}
- name: Upload RPM artifact
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ steps.rpm.outputs.rpm_file }}
path: ${{ steps.rpm.outputs.rpm_file }}
@@ -328,7 +340,7 @@ jobs:
- name: Upload nightly candidate manifest
if: ${{ steps.publish.outputs.candidate_file != '' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: nightly-candidate-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ steps.publish.outputs.candidate_file }}
@@ -360,9 +372,12 @@ jobs:
# live scripts (AppRole login, three-node Raft leader failover).
#
# GitHub-hosted ubuntu-latest, deliberately not the self-hosted sm-standard
# fleet: the HA failover script needs a working Docker daemon, and the
# self-hosted fleet is heterogeneous — a docker-dependent workflow has been
# burned by it before (see the banner in e2e-s3tests.yml, rustfs/backlog#1149).
# fleet: both lanes need a working Docker daemon, and the sm-standard-2 pool
# does not provide one. #8112 rerouted these lanes there and both died in
# seconds on "Docker is not available" while the package itself built and
# published fine, which kept the nightly chain from firing (its gate
# requires a successful build run). See also the banner in e2e-s3tests.yml
# and rustfs/backlog#1149.
kms-vault-lane:
needs: resolve-source
name: KMS live Vault lane
@@ -383,7 +398,7 @@ jobs:
NO_PROXY: 127.0.0.1,localhost
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.resolve-source.outputs.source_sha }}
@@ -473,7 +488,7 @@ jobs:
NO_PROXY: 127.0.0.1,localhost
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.resolve-source.outputs.source_sha }}
@@ -493,15 +508,17 @@ jobs:
name: Alert on scheduled failure
needs: [build, kms-vault-lane, kms-vault-ha-failover]
if: >-
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
github.repository == 'rustfs/rustfs' && (
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
)
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+6 -5
View File
@@ -32,6 +32,7 @@ concurrency:
jobs:
update-flake:
if: github.repository == 'rustfs/rustfs'
name: Update flake.lock
runs-on: ubuntu-latest
timeout-minutes: 90
@@ -39,24 +40,24 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
# persist-credentials-exempt: update-flake-lock pushes the branch and opens
# the PR. It passes FLAKE_UPDATE_TOKEN to create-pull-request itself rather
# than reusing .git/config, but that is unverified — exempt until a
# workflow_dispatch run confirms it (rustfs/backlog#1602).
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@629b284231c2a82554b724e357e47fc6020833c8 # v3
uses: DeterminateSystems/determinate-nix-action@8d87e8d5e5b8a8309d4281094560f127d9a265f1 # v3.22.5
- name: Cache Nix
uses: DeterminateSystems/flakehub-cache-action@1f9a51a2959d3e26c7838c6f3bf9f48acae525ea # v3.20.0
uses: DeterminateSystems/flakehub-cache-action@83282a8aef353db1d659d3bacf4a28b6683de0b3 # v3.22.5
- name: Check Nix flake inputs
uses: DeterminateSystems/flake-checker-action@3164002371bc90729c68af0e24d5aacf20d7c9f6 # v12
uses: DeterminateSystems/flake-checker-action@786422608c7bded2bbc9741ad9f91356842bf520 # v14
- name: Update flake.lock
id: update
uses: DeterminateSystems/update-flake-lock@fd9359ac79d0e912f1b4b947a48470b3e2799b56 # main
uses: DeterminateSystems/update-flake-lock@da03c0f078bc4b2c37ee4f7e072d34bf8f188bb3 # v29
with:
git-author-name: houseme
git-author-email: housemecn@gmail.com
+5 -5
View File
@@ -63,7 +63,7 @@ jobs:
nix-validation:
name: Nix Build & Check
if: github.event_name != 'pull_request' || github.event.action != 'closed'
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 60
permissions:
contents: read
@@ -72,12 +72,12 @@ jobs:
NIX_CURL_FLAGS: -A cargo/stable
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install Nix
uses: DeterminateSystems/determinate-nix-action@4eea0b33e3d1f02ecfe37cf16e7204c424009606 # v3.21.0
uses: DeterminateSystems/determinate-nix-action@8d87e8d5e5b8a8309d4281094560f127d9a265f1 # v3.22.5
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
extra-conf: |
@@ -86,10 +86,10 @@ jobs:
max-jobs = 1
- name: Cache Nix
uses: DeterminateSystems/flakehub-cache-action@c01e819d047464c3edf6ba778f075952af5a3aa7 # v3.21.0
uses: DeterminateSystems/flakehub-cache-action@83282a8aef353db1d659d3bacf4a28b6683de0b3 # v3.22.5
- name: Check Nix Flake Inputs
uses: DeterminateSystems/flake-checker-action@3164002371bc90729c68af0e24d5aacf20d7c9f6 # v12
uses: DeterminateSystems/flake-checker-action@786422608c7bded2bbc9741ad9f91356842bf520 # v14
with:
fail-mode: true
ignore-missing-flake-lock: false
+12 -5
View File
@@ -20,8 +20,9 @@ on:
- ".github/workflows/oidc-keycloak.yml"
- "crates/config/src/constants/oidc.rs"
- "crates/iam/src/federation/**"
- "crates/iam/src/oidc.rs"
- "crates/iam/src/oidc/**"
- "rustfs/src/admin/handlers/oidc.rs"
- "rustfs/src/admin/service/oidc_config.rs"
- "rustfs/src/admin/handlers/sts.rs"
- "scripts/test/oidc_keycloak_live.sh"
- "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
@@ -31,8 +32,9 @@ on:
- ".github/workflows/oidc-keycloak.yml"
- "crates/config/src/constants/oidc.rs"
- "crates/iam/src/federation/**"
- "crates/iam/src/oidc.rs"
- "crates/iam/src/oidc/**"
- "rustfs/src/admin/handlers/oidc.rs"
- "rustfs/src/admin/service/oidc_config.rs"
- "rustfs/src/admin/handlers/sts.rs"
- "scripts/test/oidc_keycloak_live.sh"
- "scripts/test/fixtures/keycloak-rustfs-ci-realm.json"
@@ -57,7 +59,7 @@ jobs:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -72,6 +74,11 @@ jobs:
- name: Build RustFS
run: cargo build --locked -p rustfs --bin rustfs
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install pinned request signer
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
@@ -82,7 +89,7 @@ jobs:
- name: Upload service logs
if: failure()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: oidc-keycloak-live-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-keycloak-live-*/**/*.log
@@ -101,7 +108,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
@@ -69,8 +69,7 @@ env:
jobs:
minio-source:
name: MinIO source (read-through, list-through, backfill)
# Skip on forks: needs this repository's runners and is not a contributor
# gate.
# Keep this scheduled interoperability lane scoped to the upstream repo.
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 90
@@ -93,7 +92,7 @@ jobs:
NEXTEST_LISTING: ${{ github.workspace }}/artifacts/odm-interop/minio/selection.json
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -108,11 +107,15 @@ jobs:
run: |
set -euo pipefail
mkdir -p artifacts/odm-interop/minio
docker build -t rustfs-odm-interop-minio \
-f crates/rio-v2/tests/minio_fixture_lab/Dockerfile \
crates/rio-v2/tests/minio_fixture_lab
docker run -d --name rustfs-odm-interop-minio \
--entrypoint /usr/local/bin/minio \
-e "MINIO_ROOT_USER=${MINIO_ROOT_USER}" \
-e "MINIO_ROOT_PASSWORD=${MINIO_ROOT_PASSWORD}" \
-p 9100:9000 \
minio/minio:RELEASE.2025-09-07T16-13-09Z server /data
rustfs-odm-interop-minio server /data
for _ in $(seq 1 120); do
curl -fsS http://127.0.0.1:9100/minio/health/live >/dev/null 2>&1 && break
sleep 1
@@ -166,7 +169,7 @@ jobs:
- name: Upload the MinIO interop report
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: odm-interop-minio-${{ github.run_number }}-${{ github.run_attempt }}
path: |
@@ -237,7 +240,7 @@ jobs:
- name: Checkout repository
if: steps.credentials.outputs.present == 'true'
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -286,7 +289,7 @@ jobs:
- name: Upload the ${{ matrix.provider }} interop report
if: always() && steps.credentials.outputs.present == 'true'
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: odm-interop-${{ matrix.provider }}-${{ github.run_number }}-${{ github.run_attempt }}
path: |
@@ -306,7 +309,7 @@ jobs:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+71 -26
View File
@@ -25,9 +25,10 @@
#
# Flow:
# 1. Resolve and validate the selected Build workflow run and source identity
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu) from build artifacts
# 3. Build DEB packages for amd64 and arm64
# 4. Build RPM packages for x86_64 and aarch64
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu, x86_64-musl,
# aarch64-musl) from build artifacts
# 3. Build DEB packages for amd64 and arm64, gnu and musl variants
# 4. Build RPM packages for x86_64 and aarch64, gnu and musl variants
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
@@ -76,7 +77,7 @@ jobs:
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 10
outputs:
version: ${{ steps.resolve.outputs.version }}
@@ -122,7 +123,7 @@ jobs:
semver_core='(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)'
prerelease_id='(alpha|beta|rc)\.(0|[1-9][0-9]*)'
if [[ -n "$TAG" ]]; then
if [[ "$TAG" =~ ^${semver_core}-${prerelease_id}-preview\.(0|[1-9][0-9]*)$ ]]; then
if [[ "$TAG" =~ ^${semver_core}(-${prerelease_id})?-preview\.(0|[1-9][0-9]*)$ ]]; then
BUILD_TYPE=preview
elif [[ "$TAG" =~ ^${semver_core}-${prerelease_id}$ ]]; then
BUILD_TYPE=prerelease
@@ -224,11 +225,11 @@ jobs:
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Build DEB and RPM packages for each architecture
# Build DEB and RPM packages for each architecture and libc variant
package:
name: Package (${{ matrix.arch }})
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: resolve
runs-on: ubuntu-latest
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
strategy:
fail-fast: false
@@ -236,16 +237,38 @@ jobs:
matrix:
include:
- arch: x86_64
libc: gnu
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-gnu"
# gnu binaries dynamically link glibc; musl binaries are static
# and run without a libc dependency.
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: aarch64
libc: gnu
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-gnu"
deb_depends: "libc6 (>= 2.31)"
rpm_depends: "glibc >= 2.31"
- arch: x86_64
libc: musl
deb_arch: amd64
rpm_arch: x86_64
artifact_name: "rustfs-linux-x86_64-musl"
deb_depends: ""
rpm_depends: ""
- arch: aarch64
libc: musl
deb_arch: arm64
rpm_arch: aarch64
artifact_name: "rustfs-linux-aarch64-musl"
deb_depends: ""
rpm_depends: ""
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -256,17 +279,17 @@ jobs:
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
SOURCE_VERSION: ${{ needs.resolve.outputs.version }}
DEV_SEQUENCE: ${{ needs.resolve.outputs.dev_sequence }}
DEB_ARCH: ${{ matrix.deb_arch }}
RPM_ARCH: ${{ matrix.rpm_arch }}
ARCH: ${{ matrix.rpm_arch }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
normalized=$(./scripts/release/package_versions.sh \
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$DEB_ARCH" "$RPM_ARCH")
"$BUILD_TYPE" "$SOURCE_VERSION" "$DEV_SEQUENCE" "$ARCH" "$LIBC")
printf '%s\n' "$normalized" >> "$GITHUB_OUTPUT"
- name: Download binary artifact from build run
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: ${{ matrix.artifact_name }}*
path: ./binary-artifact
@@ -308,6 +331,8 @@ jobs:
DEB_VERSION: ${{ steps.versions.outputs.deb_version }}
DEB_ARCH: ${{ matrix.deb_arch }}
DEB_FILE: ${{ steps.versions.outputs.deb_file }}
DEB_DEPENDS: ${{ matrix.deb_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
@@ -345,9 +370,17 @@ jobs:
Section: utils
Priority: optional
Architecture: ${DEB_ARCH}
Depends: libc6 (>= 2.31)
EOF
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
if [[ -n "$DEB_DEPENDS" ]]; then
echo "Depends: ${DEB_DEPENDS}" >> "${PKG_DIR}/DEBIAN/control"
fi
cat >> "${PKG_DIR}/DEBIAN/control" << EOF
Maintainer: RustFS Team <support@rustfs.com>
Description: High-performance distributed object storage
Description: High-performance distributed object storage (${LIBC} build)
RustFS is a high-performance distributed object storage software
built using Rust. It is compatible with MinIO and S3 API.
Homepage: https://rustfs.com
@@ -389,15 +422,24 @@ jobs:
RPM_RELEASE: ${{ steps.versions.outputs.rpm_release }}
RPM_ARCH: ${{ matrix.rpm_arch }}
RPM_FILE: ${{ steps.versions.outputs.rpm_file }}
RPM_DEPENDS: ${{ matrix.rpm_depends }}
LIBC: ${{ matrix.libc }}
run: |
set -euo pipefail
echo "Building RPM for ${RPM_ARCH}"
echo "Building RPM for ${RPM_ARCH} (${LIBC})"
sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential rpm
sudo gem install fpm
./scripts/test_package_versions.sh --require-package-managers
# gnu binaries dynamically link glibc and must declare it; musl
# binaries are statically linked and need no libc dependency.
depends_args=()
if [[ -n "$RPM_DEPENDS" ]]; then
depends_args+=(--depends "$RPM_DEPENDS")
fi
# Create config file for fpm (DEB build creates it in its package dir structure,
# but fpm needs the file to exist before packaging)
mkdir -p ./tmp-pkg/etc/default
@@ -420,9 +462,9 @@ jobs:
--iteration "$RPM_RELEASE" \
--architecture "$RPM_ARCH" \
--package "$RPM_FILE" \
--depends "glibc >= 2.31" \
"${depends_args[@]}" \
--description "High-performance distributed object storage (${LIBC} build)" \
--maintainer "RustFS Team <support@rustfs.com>" \
--description "High-performance distributed object storage" \
--url "https://rustfs.com" \
--license "Apache-2.0" \
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
@@ -453,16 +495,16 @@ jobs:
echo "✅ RPM built: $RPM_FILE"
- name: Upload packages to artifacts
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: packages-${{ matrix.arch }}
name: packages-${{ matrix.arch }}-${{ matrix.libc }}
path: |
*.deb
*.rpm
retention-days: 30
- name: Upload packages to Cloudflare R2
if: env.R2_ACCESS_KEY_ID != ''
if: needs.resolve.outputs.build_type != 'preview' && env.R2_ACCESS_KEY_ID != ''
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
@@ -551,10 +593,11 @@ jobs:
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# GitHub stores release asset names with '~' normalized to '.'
# (e.g. rustfs_1.0.0~rc.2_amd64.deb is stored as
# rustfs_1.0.0.rc.2_amd64.deb), so checksum entries must
# reference the name as stored on the release.
# GitHub stores release asset names with '~' normalized to '.'.
# Current package file names embed the raw tag and contain no
# '~', but older releases may still ship such assets, so the
# checksum entries must reference the name as stored on the
# release.
github_base="${base//\~/.}"
# Remove any stale entry (both naming variants), then append
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
@@ -575,7 +618,9 @@ jobs:
summary:
name: Summary
needs: [ resolve, package ]
if: always()
# Every completed Build and Release run triggers this workflow; when
# resolve declined it (not a release tag), there is nothing to summarise.
if: always() && needs.resolve.result != 'skipped'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
+9 -6
View File
@@ -49,6 +49,7 @@ env:
jobs:
warp-ab:
if: github.repository == 'rustfs/rustfs'
name: Warp A/B budget gate
runs-on: sm-standard-2
# A normal nightly restores the last successful binary and builds only the
@@ -57,7 +58,7 @@ jobs:
timeout-minutes: 180
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0 # baseline may be an earlier successful scheduled head
@@ -94,7 +95,7 @@ jobs:
- name: Find last successful scheduled baseline
id: scheduled_baseline
if: github.event_name == 'schedule'
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
result-encoding: string
script: |
@@ -271,7 +272,7 @@ jobs:
- name: Upload A/B results
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: hotpath-warp-ab-${{ github.run_number }}
# Includes per-cell median_summary.csv / baseline_compare.csv, both gates,
@@ -364,15 +365,17 @@ jobs:
# composite action already reports cancelled/timed-out jobs in the issue
# body.
if: >-
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
github.repository == 'rustfs/rustfs' && (
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
)
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
+5 -4
View File
@@ -42,12 +42,13 @@ concurrency:
jobs:
check-ephemerality:
if: github.repository == 'rustfs/rustfs'
name: Check runner ephemerality
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -65,14 +66,14 @@ jobs:
# Same ci-8 mechanism as coverage.yml, audit.yml and the nightly lanes:
# scheduled runs file a tracking issue, manual dispatch stays quiet so
# debugging never produces a spurious alert.
if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
@@ -12,7 +12,7 @@
# assert the expected outage
# M-<topo>EC<P>-rec after restoring every node, reads/writes succeed
#
# 10 topologies x EC levels = 38 combinations, 320 cases. The full sweep runs
# 9 topologies x EC levels = 36 combinations, 300 cases. The full sweep runs
# 3-5 hours and is dispatched on demand (e.g. after a nightly chain goes
# green), not nightly. Case ids and expectations come from product source
# (default_parity_count, erasure set sizing) via rustfs/auto-testing.
@@ -32,7 +32,7 @@ on:
default: main
type: string
matrix_topo:
description: 'Topology filter like 4x2; empty sweeps all ten topologies'
description: 'Topology filter like 4x2; empty sweeps all nine topologies'
required: false
default: ''
type: string
@@ -95,7 +95,7 @@ jobs:
- name: Checkout auto-testing scripts
timeout-minutes: 5
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ inputs.auto_testing_ref }}
@@ -174,12 +174,12 @@ jobs:
} > "${REPORT_FILE}"
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
# An unfiltered sweep must land exactly 320 cases (38 topology x EC
# An unfiltered sweep must land exactly 300 cases (36 topology x EC
# combinations at 2*(nodes+1) cases each); filtered sweeps are held
# to internal consistency. Missing/harness-error rows always fail.
EXPECTED_TOTAL=""
if [ -z "${{ inputs.matrix_topo }}" ] && [ -z "${MATRIX_EC}" ]; then
EXPECTED_TOTAL=320
EXPECTED_TOTAL=300
fi
jq -e --arg testing_sha "${AUTO_TESTING_SHA}" --argjson expected_total "${EXPECTED_TOTAL:-null}" '
.schema_version == 1 and
@@ -204,7 +204,7 @@ jobs:
- name: Upload test logs & evidence
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-fault-tolerance-matrix-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -83,7 +83,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -113,7 +113,7 @@ jobs:
- name: Checkout auto-testing scripts
timeout-minutes: 5
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || inputs.auto_testing_ref || 'main' }}
@@ -294,7 +294,7 @@ jobs:
- name: Upload test logs & evidence
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -386,7 +386,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}/fault-tolerance.json
@@ -42,13 +42,13 @@ concurrency:
jobs:
prepare:
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.event == 'schedule' }}
runs-on: ubuntu-latest
runs-on: smoke-testing
timeout-minutes: 10
outputs:
manifest: ${{ steps.candidate.outputs.manifest }}
performance_ready: ${{ steps.perf_probe.outputs.performance_ready }}
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Resolve published candidate
@@ -63,7 +63,7 @@ jobs:
CHAIN_OUTPUT: ${{ runner.temp }}/chain-candidate.json
run: python3 scripts/resolve_functional_candidate.py
- name: Retain candidate identity
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-candidate-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-candidate.json
@@ -188,16 +188,16 @@ jobs:
complete-chain:
needs: [prepare, upgrade, s3, kms, tier, storage, heal, pool, security, replication, fault-tolerance, table, performance]
if: ${{ always() && needs.prepare.result != 'skipped' }}
runs-on: ubuntu-latest
runs-on: smoke-testing
timeout-minutes: 10
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download suite evidence
id: download
continue-on-error: true
uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: functional-chain-*-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-evidence
@@ -213,7 +213,7 @@ jobs:
--output "${RUNNER_TEMP}/chain-report.json"
- name: Retain chain report regardless of test verdict
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-report-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -231,7 +231,7 @@ jobs:
--directory "${RUNNER_TEMP}/chain-evidence"
--output "${RUNNER_TEMP}/chain-complete.json"
- name: Upload complete-chain evidence
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-complete-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-complete.json
+4 -4
View File
@@ -66,7 +66,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -93,7 +93,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -366,7 +366,7 @@ jobs:
- name: Upload test logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-heal-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -467,7 +467,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-heal-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-heal-${{ github.run_id }}-${{ github.run_attempt }}/heal.json
+4 -4
View File
@@ -59,7 +59,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -85,7 +85,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -315,7 +315,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-kms-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -414,7 +414,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-kms-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-kms-${{ github.run_id }}-${{ github.run_attempt }}/kms.json
@@ -90,10 +90,10 @@ env:
jobs:
runner-check:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
runs-on: ubuntu-latest
runs-on: sm-standard-2
timeout-minutes: 5
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Check performance runner before entering its queue
@@ -116,7 +116,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -144,7 +144,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -307,7 +307,7 @@ jobs:
- name: Upload test logs & results
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-perf-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -349,7 +349,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-performance-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-performance-${{ github.run_id }}-${{ github.run_attempt }}/performance.json
+31 -7
View File
@@ -7,6 +7,11 @@ on:
description: Verified candidate and chain attempt from the chain driver
type: string
required: true
pool_timeout_minutes:
description: Wall-clock budget (minutes) for the pool run step
type: string
required: false
default: '240'
workflow_dispatch:
inputs:
rustfs_version:
@@ -39,6 +44,10 @@ on:
description: 'Run the pool decommission step (3-pool topology only)'
type: boolean
default: true
pool_timeout_minutes:
description: 'Wall-clock budget (minutes) for the whole pool run: fill + rebalance wait + decommission'
required: false
default: '240'
cleanup_before:
description: 'Reset the nodes before the test (DESTROYS existing data/config)'
type: boolean
@@ -82,7 +91,8 @@ jobs:
pool-expansion-test:
name: Pool expansion / decommission test
runs-on: smoke-testing
timeout-minutes: 60
# Leave setup and finalizer headroom beyond the default 240-minute test.
timeout-minutes: 360
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
env:
RUSTFS_POOL_ADMIN_ENDPOINT: ${{ secrets.RUSTFS_POOL_ADMIN_ENDPOINT || vars.RUSTFS_POOL_ADMIN_ENDPOINT || 'http://rustfs-node1:9000' }}
@@ -92,7 +102,7 @@ jobs:
RUSTFS_POOL_NODE_ENDPOINTS: ${{ secrets.RUSTFS_POOL_NODE_ENDPOINTS || vars.RUSTFS_POOL_NODE_ENDPOINTS || 'http://rustfs-node1:9000 http://rustfs-node2:9000 http://rustfs-node3:9000 http://rustfs-node4:9000' }}
steps:
- name: Checkout chain tooling
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -104,7 +114,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -251,7 +261,15 @@ jobs:
- name: Run pool expansion & decommission test
id: pool_test
timeout-minutes: 45
# The suite budgets up to 24h each for rebalance and decommission
# (REBALANCE_TIMEOUT / DECOMMISSION_TIMEOUT in rustfs_pool_expand.sh);
# the previous 45m step cap killed four consecutive runs mid-rebalance
# with steps 1-7 green. 240m covers the observed pace (rebalance 1/4
# in ~21m) plus one decommission pass; override per dispatch.
# fromJSON coerces the string dispatch input to the number that
# timeout-minutes requires; the bare || form yielded the string '240'
# and the runner rejected the template (Unexpected value '240').
timeout-minutes: ${{ fromJSON(inputs.pool_timeout_minutes || '240') }}
# Case failures keep the run green: the report and the backlog
# issue manager carry the product signal.
continue-on-error: true
@@ -261,7 +279,9 @@ jobs:
if [ "${{ inputs.pools || '3' }}" = "3" ]; then
STEPS="$STEPS,7,8"
if [ "${{ inputs.run_decommission != 'false' }}" = "true" ]; then
STEPS="$STEPS,9"
# Step 10 asserts the surviving pools after the removal
# (rustfs#8726), so it only runs together with step 9.
STEPS="$STEPS,9,10"
fi
fi
ARGS=(--steps "$STEPS" --with-warp -y \
@@ -275,6 +295,10 @@ jobs:
if [ -n "${RUSTFS_POOL_PROXY_ENDPOINT}" ]; then
ARGS+=(--proxy-endpoint "${RUSTFS_POOL_PROXY_ENDPOINT}")
fi
# Finalize the removal so step 10 can exercise the surviving
# pools; passing it without step 9 selected would run step 10
# against a topology where nothing was removed.
case ",${STEPS}," in *,9,*) ARGS+=(--finalize-decommission) ;; esac
if [ -n "${{ inputs.package_url }}" ]; then
ARGS+=(--package-url "${{ inputs.package_url }}")
elif [ -n "${{ inputs.rustfs_version }}" ]; then
@@ -656,7 +680,7 @@ jobs:
- name: Upload test logs
timeout-minutes: 2
if: always()
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-pool-test-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/rustfs-pool-${{ github.run_id }}-${{ github.run_attempt }}
@@ -764,7 +788,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-pool-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-pool-${{ github.run_id }}-${{ github.run_attempt }}/pool.json
@@ -71,7 +71,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -97,7 +97,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -323,7 +323,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-replication-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -419,7 +419,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-replication-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-replication-${{ github.run_id }}-${{ github.run_attempt }}/replication.json
+4 -4
View File
@@ -47,7 +47,7 @@ jobs:
if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
steps:
- name: Checkout repository (for report parser)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
@@ -73,7 +73,7 @@ jobs:
run: python3 scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -292,7 +292,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-s3-compat-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -391,7 +391,7 @@ jobs:
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-s3-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-s3-${{ github.run_id }}-${{ github.run_attempt }}/s3.json
+6 -6
View File
@@ -87,7 +87,7 @@ jobs:
# the workspace root would wipe the auto-testing clone above (that is
# exactly how run 33934141181 lost rustfs-security-test.sh).
- name: Checkout repository (for the OIDC live gate script)
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
path: rustfs-repo
@@ -106,10 +106,10 @@ jobs:
if: ${{ inputs.chain_manifest != '' }}
env:
CHAIN_MANIFEST: ${{ inputs.chain_manifest }}
run: python3 scripts/functional_chain_evidence.py consume
run: python3 rustfs-repo/scripts/functional_chain_evidence.py consume
- name: Checkout auto-testing scripts
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
@@ -325,7 +325,7 @@ jobs:
- name: Upload report and logs
timeout-minutes: 2
if: ${{ always() && steps.evidence.outcome == 'success' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-security-test-${{ github.run_id }}-${{ github.run_attempt }}
path: |
@@ -389,13 +389,13 @@ jobs:
CHAIN_TEST_OUTCOME: ${{ steps.test.outcome }}
CHAIN_REPORT_OUTCOME: ${{ steps.report.outcome }}
run: >-
python3 scripts/functional_chain_evidence.py record --suite security
python3 rustfs-repo/scripts/functional_chain_evidence.py record --suite security
--report "${SECURITY_ARTIFACTS_DIR}/suite-report.md"
--output "${RUNNER_TEMP}/chain-security-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}/security.json"
- name: Upload chain evidence
if: ${{ always() && steps.chain_record.outputs.written == 'true' }}
uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: functional-chain-security-${{ github.run_id }}-${{ github.run_attempt }}
path: ${{ runner.temp }}/chain-security-${{ github.run_id }}-${{ github.run_attempt }}/security.json

Some files were not shown because too many files have changed in this diff Show More