Compare commits

..

77 Commits

Author SHA1 Message Date
overtrue c4a7423f99 fix(ecstore): create parents before admitted truncate opens
Align the heal tests with removal-disabled preservation and local-only outcome
fields, and refresh the e2e-full selection for the synced tests.
2026-10-31 08:38:51 +08:00
overtrue 2db4ed244a fix(ecstore): scope the merged direct-writer test hook to test builds 2026-10-31 06:40:49 +08:00
overtrue 93a61df1cf chore(gateway): merge main into integration
Keep native publication recording in main's native_io helpers and resource
writers, and adapt merged tests to the owned Body, Bytes write input, and
fallible find_disk.
2026-10-31 06:15:49 +08:00
overtrue feec0e9e4a test(s3select): use the owned JSON input types in merged planner tests 2026-10-31 03:08:13 +08:00
overtrue 38baeeb11d chore(gateway): merge main into integration
Record native publications on the paths main's commit-fusion work opened
directly: the lazy truncate writer and the fused delete-recovery activation
and receipt renames. Keep extract_params_header in the shared audit builder.
2026-10-31 01:40:30 +08:00
overtrue 1af039c972 chore(gateway): merge main into integration 2026-10-30 20:14:52 +08:00
overtrue a2fc7c13b4 chore(gateway): merge main into integration 2026-10-30 18:41:32 +08:00
overtrue dbe01e5cb1 chore(gateway): raise test-count rows for tests merged from main 2026-10-30 14:35:18 +08:00
overtrue ffc27fa6db chore(gateway): merge main into integration 2026-10-30 14:29:25 +08:00
overtrue 7e8edab0d3 chore(gateway): merge main into integration 2026-10-30 11:23:05 +08:00
overtrue 22057ad4c5 chore(gateway): merge main into integration 2026-10-30 07:08:33 +08:00
overtrue 279a88ab56 chore(gateway): merge main into integration 2026-10-29 20:19:36 +08:00
overtrue b4abe64ada chore(gateway): merge main into integration
Sync main 84e24e767 into gateway/integration. Ports main-side additions to
the integration boundaries: the tier deletion diagnostics admin handlers move
to the owned admin request/response types, and the new lifecycle rule checks
in ecstore use the persisted lifecycle types instead of s3s DTOs. Keeps the
gateway git source in deny.toml while dropping the smithy-rs pin main retired,
and refreshes the test-count baseline for the merged tree.
2026-10-29 16:01:13 +08:00
overtrue add363f084 chore(gateway): merge main into integration
Sync main 9601a7c4b into gateway/integration. Ports main-side additions to the
owned admin boundary (plugin delegation/attempt handlers, notification target
edit tests), drops a new ecstore s3s::dto reference in favour of the persisted
lifecycle filter, and refreshes the e2e selection digests and test-count
baseline for the merged tree.
2026-10-29 07:31:24 +08:00
overtrue 1f2dbe36e5 test(ecstore): pin native history on cancellation under the cancellable queue
main moved the local write path onto the disk blocking queue, which skips an
operation once its waiter is dropped, so a borrowed write cancelled while it is
queued no longer creates its file. The integration-only native publication
test still expected the queued open to run. Assert the new execution model
instead (the open never runs) while keeping its safety claims: the Started
record is taken before queueing and the cancelled publication never regrants
native history.
2026-10-27 22:08:04 +08:00
overtrue 67fbafa13e chore(gateway): merge main into integration
Source: main 4e48c390e7
Target: gateway/integration 1000860cb8

Conflicts resolved, keeping both sides:
- crates/ecstore/src/disk/{fs,local}.rs: main's batched rollback writes,
  unified blocking write path and relocated open_file, with integration's
  native publication records restored at each publication point and the
  publication_history field kept beside main's multipart proof authority.
- crates/ecstore/src/set_disk/ops/multipart.rs: union of the imports.
- rustfs/src/storage/rpc/node_service.rs: main's MRF lifecycle reader
  capability branch; the heal capability keeps integration's restore
  copy-back probe.
- rustfs/src/admin/handlers/{audit,event,plugins_instances}.rs: main's
  target_read inventory snapshot on integration's owned request boundary;
  the new event.rs inventory test now builds owned requests.
- scripts/check_s3s_footprint.sh: integration's baseline-file version.
- .config/e2e-{full,smoke}-selection.txt: digests recomputed over the
  union of both heads' CI listings (full 700 darwin / 708 linux, smoke 232).

Also migrated main's new rebalance delete-all expiry test to the persisted
lifecycle types integration uses in ecstore, and raised the test-count
baseline to the merged counts (every decrease comes from main).
2026-10-27 08:38:17 +08:00
Chris An 1000860cb8 refactor(admin): drop the legacy target descriptor JSON response helper (#8968)
`target_descriptor::build_json_response` returned the s3s response and
had no production caller left once the event, audit and plugin instance
handlers moved to `owned_build_json_response`; it survived only for the
legacy/owned equivalence test. Remove it, give the owned helper its own
documentation, and make the test assert the owned response directly:
status, `application/json`, the echoed `x-request-id` when present and
only then, and the body bytes. The file no longer imports s3s, so the
footprint baseline drops from 66 to 65 files.
2026-10-27 00:15:37 +08:00
Chris An b783f404ad refactor(admin): migrate identity and cluster handlers to owned requests (#8966)
* refactor(admin): migrate identity and cluster handlers to owned requests

Move the T1.12 batch 4 admin handlers (user, service_account, oidc, tier,
system, pools, quota, kms_dynamic, kms_backup, rebalance, module_switch,
bucket_meta, cluster_snapshot, realtime, bucket_recovery, profile) from the
s3s request/response types to the owned AdminRequest/AdminResponse
boundary. All 90 impl Operation blocks now take AdminRequest and return
AdminResponse; handler logic, authorization order, error codes and response
bytes are unchanged. The handlers use the shared owned helpers
(owned_json_response, owned_admin_json_response, authorize_profile_envelope,
the envelope-form runtime sources and audit constructors) instead of local
copies.

profile.rs drops the s3s forms of authorize_profile_request and
profile_not_implemented_response now that profile_admin.rs is on the
envelope gate too; profile_not_implemented_response answers with an
AdminResponse. The realtime metrics stream moves from the s3s ByteStream
adapter to Body::from_stream, and the rebalance idle-404 document still
renders through s3s::S3Error::to_http_response with its body converted at
the edge.

The idp_compat delegators ImportIamV2 and AddServiceAccountLdap move to the
owned trait as well: with their targets on both traits, the method-call
delegation became ambiguous between the owned impl and the router blanket
impl, so they now take the owned request directly.

KmsAdminAudit::from_request loses its last production caller with this batch
and is deleted; its comparison test now asserts the envelope facts directly.

Lower the s3s footprint baseline to the measured 68 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): remove the legacy admin helpers batch 4 superseded

Batch 4 moved the last production callers of the s3s-typed admin helpers
to their owned forms, which left `admin_json_response`, `json_response`,
`authenticate_request`, `authorize_admin_request`, `object_store_from_req`
and `object_store_from_extensions` unused, along with the `Body`,
`S3Request` and `S3Response` re-exports of the admin `s3` facade. Remove
them. The owned forms keep their documentation; the behavior tests of the
removed `json_response` now exercise `owned_json_response`, and the
compat-prefix test asserts the owned response directly.

Two tests existed only to compare a removed legacy helper with its owned
twin (`owned_json_response_matches_the_legacy_shape`) or to cover the
removed legacy gate (`authorize_admin_request_without_credentials_is_rejected`);
they go with the helpers, and the owned equivalents stay. The rustfs
test-count row drops from 5957 to 5955 and the s3s footprint from 68 to 66
files.
2026-10-26 22:23:47 +08:00
Chris An 11a991c777 refactor(admin): complete the owned admin request boundary (#8949)
* refactor(admin): complete the owned admin request boundary

Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.

`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.

No handler behavior changes; the remaining migrations follow in batches.

* chore(admin): add the T1.12 admin handler codemod

The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.

* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)

Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.

The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.

Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* refactor(admin): add owned forms of the shared admin response helpers

Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.

* refactor(admin): use the shared owned response helpers in batch 2 (#8954)

Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.

Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.

Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).

* fix(admin): preserve legacy body limit error responses

* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)

Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.

The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.

object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.

The s3s footprint baseline drops from 99 to 90 files.

* test(admin): anchor the replication diff body contract on the call

The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.

* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)

* fix(admin): preserve catalog body limit response bytes
2026-10-26 17:04:02 +08:00
Chris An 3f2b6f690a refactor(admin): migrate health and prefix usage to owned requests (#8887)
* refactor(admin): add owned health router boundary

* refactor(admin): migrate prefix usage to owned request
2026-10-26 07:34:31 +08:00
overtrue 88b626e7a0 chore(gateway): merge main into integration
Source: main 0f24511535
Target: gateway/integration 8cf6187ff4

Resolve the e2e-full selection digest conflict by recomputing both
platform digests over the merged nextest listing (685 darwin / 693
linux): main adds two rpc_fault_proxy unit tests, integration keeps its
dual-stack differential and anonymous-access additions. Raise the rustfs
test-count baseline row to the merged count (5932).
2026-10-25 09:58:02 +08:00
overtrue 8cf6187ff4 chore(gateway): merge main into integration 2026-10-25 01:50:03 +08:00
Chris a0a0cee9eb refactor(auth): derive policy conditions from AuthzFacts (#8442)
* refactor(auth): extract authorization facts

* test(e2e): update smoke selection baseline

* test(e2e): update full selection baseline
2026-10-24 16:56:01 +09:00
overtrue 80fff4b5c5 Merge gateway/integration (#8844) into the main sync
Only the s3s footprint baseline conflicted; s3_error_lines tightens to 289.
2026-10-24 09:30:22 +08:00
Chris d91dfcedd2 refactor(s3): migrate private multipart validation errors (#8844) 2026-10-24 09:17:48 +09:00
overtrue 79c4fedf8c Merge gateway/integration into the main sync
Brings in #8825, #8831, #8836 and #8839. Conflicts were import lists and test blocks (both sides kept), plus the new restore_copyback request flag next to main's data-movement field. Main replaced completed_multipart_object_part with prepare_complete_multipart_parts, so the stale import is dropped.

Main's xl.meta fast-path scanner (inspect_v2_object_slice) predates #8836 and classified a transitioned version with a pending v1 restore copyback as having no physical data dir, which made find_unshared_data_dir_for_version drop the copyback data dir. The scanner now reads the copyback marker with get_consistent_bytes semantics (both aliases, ASCII case-insensitive, empty or conflicting values count as absent), and uses_data_dir keeps main's allocation-free transition check plus #8836's copyback exclusion. find_unshared_data_dir_for_version_keeps_pending_copyback_data_dir failed on the plain merge and passes now.
2026-10-24 07:51:55 +08:00
Chris 73c846e09c refactor(storage): migrate option errors to domain types (#8839) 2026-10-24 05:50:10 +09:00
Chris 6938c467d0 fix(ecstore): finalize restore metadata after local copy-back (#8836)
* fix(ecstore): finalize restore metadata after local copy-back

* test(ecstore): ratchet added restore finalization cases

* fix(ecstore): keep restore copyback compatible with older peers

* fix(ecstore): gate restore copyback on fleet compatibility

* fix(ecstore): retain transition identity during restore copyback

* fix(ecstore): route committed restore copyback locally

* fix restore copyback cleanup

* fix(ecstore): honor restore probe test fallback
2026-10-24 02:34:29 +09:00
Chris 5323df7049 fix(protocols): clear clippy failures after T1.5 merge (#8831)
fix(protocols): clear clippy failures after merge
2026-10-23 17:37:26 +09:00
Chris 55b6fbd260 refactor(protocols): move StorageBackend to gateway types (#8825)
* refactor(protocols): move storage backend to gateway types

* test: update e2e test count baseline

* fix: preserve copy source keys and unknown lengths

* test: update protocols test count baseline

* fix(protocols): preserve typed SFTP backend errors

* test(protocols): use gateway list input in policy tests
2026-10-23 15:46:28 +09:00
overtrue dd677cc121 Merge main into gateway/integration
Picks up main through 44a73b0be. Conflicts: rustfs/src/error.rs keeps the rustfs-s3-types ApiError conversion and drops the helper main's #8816 replaced; scripts/check_s3s_footprint.sh keeps the integration script, whose baseline lives in .config (main only retuned a constant the integration script does not have). Main's new tests use the rustfs-s3-types error code in admin replication and reach s3s::BodySizeLimitExceeded through the app's test-only s3 facade, so the s3s file count stays at 118; s3_error_lines tightens to 302. The test-count baseline is regenerated; the e2e selections are unchanged.
2026-10-23 08:40:02 +08:00
overtrue 0c2f327974 Merge gateway/integration into the main sync
Brings in #8812 and #8818 from gateway/integration. #8812 predates main's heal error classifier (#8778); its two retryable variants (TooManyOpenFiles, OutdatedXLMeta) move into the classifier's transient arms for storage and disk errors, and its test is kept.
2026-10-23 07:22:20 +08:00
overtrue df42cf1ce8 test(e2e): use a non-default region for the differential negative control
Since main's region-aware CreateBucket and GetBucketLocation (#8739, #8743), us-east-1 answers an empty LocationConstraint and 200 for an owned re-create, so the control no longer reported its own region. Also refresh the e2e-full linux selection from the CI listing (682 tests).
2026-10-23 07:21:19 +08:00
Chris d290d46b34 test: reserve nextest capacity for progress probes (#8818) 2026-10-23 06:12:31 +09:00
Chris 0d297ed8cd fix(heal): retry transient metadata conflicts (#8812)
* fix(heal): retry transient metadata conflicts

* chore: update heal test count baseline
2026-10-23 01:47:55 +09:00
overtrue b8dffa2f1d Merge main into gateway/integration
Picks up the 20 main commits since 62b6f5f3f. The one conflict is the tier free-version recovery test setup: it keeps the integration-only fixture isolation (background recovery disabled while the fixture starts) and main's mock tier registration (#8804 skips scans with no tier). The e2e selections are unchanged by these commits; the test-count baseline is regenerated.
2026-10-22 20:17:09 +08:00
overtrue 62b6f5f3fc Merge main into gateway/integration
The scheduled sync has stopped on conflicts since 2026-10-19. Resolved by keeping both sides: main's restructured delete-rollback path in disk/local.rs with the integration-only native-publication record, main's per-instance region with the rustfs-s3-types Region, main's new lifecycle, rebalance and admin code ported to the persisted lifecycle types and rustfs-s3-types errors, both sides' new scanner tests, and the union of the access_consumer imports. The e2e smoke, full (darwin) and repl-nightly selections and the test-count baseline are regenerated from the merged tree.
2026-10-22 10:14:41 +09:00
Chris be0986115c refactor(s3select): run queries on gateway DTO request shapes (#8794)
* refactor(s3select): run queries on an engine request built from gateway DTO shapes

Part of T1.5 (rustfs/backlog#2746), batch 2.

* fix(s3select): keep the select request conversion in the app layer

The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.
2026-10-22 08:16:48 +09:00
Chris 1bd91c8dad refactor(s3-client): move remote error codes and DTO enums off s3s (#8783)
Part of T1.5 (rustfs/backlog#2746), batch 1.
2026-10-22 06:15:04 +09:00
Chris dcb22bd4d5 refactor(ecstore): move bucket configs onto gateway persisted types (#8709)
* refactor(ecstore): move bucket configs onto gateway persisted types

Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.

- ecstore: BucketConfigXml is implemented by the gateway codec for every
  family; metadata, metadata_sys, versioning, object lock, replication and
  lifecycle consumers take the persisted types. Region moves to
  rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
  rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
  still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
  persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
  persisted types in storage/storage_api.rs by writing one codec's XML and
  reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.

The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.

* test(app): prepare the PutBucketEncryption config from the request

The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.

* style(app): format the PutBucketEncryption test
2026-10-21 23:51:23 +09:00
Chris fa17718444 refactor(admin): move error types to rustfs-s3-types (#8440)
* refactor(protocols): move error types to rustfs-s3-types by codemod

T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).

* refactor(admin): move error types to rustfs-s3-types (WIP, no build)

T1.2 batch 4 (rustfs/backlog#2743), work in progress: rustfs check
still fails with 16 errors (listed in the PR body).

- codemod: --facade PATH treats a crate-local re-export of s3s's error
  items like s3s for the four names (9 new self-tests, 9 mutants red).
- codemod output for rustfs/src/admin and rustfs/src/site_replication
  (--facade crate::storage_api::site_replication::s3); 6 glob children
  get an explicit rustfs_s3_types::s3_error import.
- admin/router.rs: the Operation trait returns rustfs_s3_types::S3Result.
- error.rs: From<ApiError> builds the rustfs-s3-types error; the s3s
  conversion is derived from it.

* fix(admin): convert error types at the s3s boundary

Batch 4 of T1.2 left the admin and site_replication modules returning
rustfs_s3_types::S3Error while their callers and callees on the storage
and auth side still speak s3s::S3Error. Convert at each edge with the
existing From impls in rustfs-s3-types (compat_s3s.rs) instead of
changing any code, message or status:

- admin/auth.rs, admin/handlers/object_zip_download.rs,
  admin/handlers/table_catalog/mod.rs: map the s3s result returned by
  check_key_valid / authorize_request / authorize_internal_object_request
  into the admin error type with .map_err(Into::into).
- storage/ecfs.rs: the s3s-facing side maps the admin site_replication
  gate results the other way.
- admin/handlers/pools.rs: ApiError.code stays s3s::S3ErrorCode, so the
  rustfs code is converted with .into() when the struct is built, in
  production and in the two tests that build or compare it.
- admin/handlers/heal.rs, admin/runtime_sources.rs: the router returns
  s3s errors (S3Route impl); the expected rustfs code is converted with
  .into() before the comparison.
- admin/handlers/table_catalog/mod.rs: iceberg_rest_error takes
  &'static str so the Cow<'static, str> custom code borrows it; every
  caller already passes an ICEBERG_ERROR_* constant or a &'static str.
- admin/handlers/quota.rs: drop a now same-type .into() after s3_error!.
- storage_api.rs, site_replication/mod.rs: drop re-exports and an import
  that no longer have users (clippy -D warnings).

cargo check -p rustfs --all-targets and cargo clippy -p rustfs
--all-targets -- -D warnings are clean; the s3s footprint baseline is
unchanged at files=172, s3_error_lines=303, ecstore_files=28.

Refs rustfs/backlog#2743, rustfs/backlog#2734.
2026-10-21 22:50:27 +09:00
overtrue 99abffc57a refactor(lifecycle): replace s3s domain types with Gateway DTOs
Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.

Linux test validation before the final import-path-only adjustment:

cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.

cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.

cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.

Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.

cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.

Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.
2026-10-17 07:03:41 +08:00
overtrue fc061c7930 fix(ecstore): retry tier commit intent lock contention
Retry transient namespace lock timeouts within the existing three-attempt
CAS budget. Reload and verify the original mutation identity after
contention, and preserve the final typed error when attempts are exhausted.
Cover save/read contention, exhausted retries, replacement identities and
CAS reconciliation without changing the existing timeout or lock policy.

Validation:
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(tier_mutation)' --test-threads 1 --no-fail-fast: 52 passed.
- cargo nextest run --locked -p rustfs-ecstore --features test-util --lib -E 'test(intent_advance_) | test(=services::tier::tier::tests::caller_cancellation_after_durable_prepare_does_not_hide_the_mutation)' --test-threads 1 --no-fail-fast: 5 passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings: passed.
- cargo clippy --locked -p rustfs-ecstore --all-targets --features test-util -- -D warnings: passed.
- 27 mutation pairs failed at the intended assertions and passed after restoring the production code.
- bash scripts/check_test_count_ratchet.sh --update, then bash scripts/check_test_count_ratchet.sh: 52 crates, 22013 tests, 165 ignored; ecstore increases from 6581 to 6585.
2026-10-17 01:19:11 +08:00
overtrue d86bf1360f fix(ecstore): recover delete batches with missing versions
Treat an explicit non-null version as a no-op when it is absent from both recovery sources. Keep rejecting missing null versions, delete-marker mismatches and conflicting local generations.

Add returning-disk and partial-receipt regressions, extend journal boundary checks, and regenerate the ecstore test-count baseline. Production faults exercise the 24 added assertion sites and 7 semantic checks: 33 faults failed at the expected sites and passed after full restoration.

Validation (LANG=en_US.UTF-8, one build job):

cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(delete_recovery) | test(quorum_delete_cleanup_coordinator_binds_single_and_batch) | test(crash_delete_retry_batch_reclaims_inline_and_reserved_external)' --test-threads 1 --no-fail-fast
Summary [  34.580s] 69 tests run: 69 passed, 6058 skipped; exit 0

cargo clippy --locked -p rustfs-ecstore --all-targets -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 5m 37s; exit 0

All 19 final Quick Checks completed with exit 0:

- shellcheck --version && actionlint
- cargo fmt --all --check
- ./scripts/check_unsafe_code_allowances.sh
- ./scripts/check_layer_dependencies.sh
- ./scripts/check_architecture_migration_rules.sh
- ./scripts/check_logging_guardrails.sh
- ./scripts/check_error_other_format_ratchet.sh
- ./scripts/check_no_tokio_io_uring.sh
- ./scripts/check_extension_schema_boundaries.sh
- ./scripts/check_body_cache_whitelist.sh
- ./scripts/check_s3s_footprint.sh
- ./scripts/check_test_count_ratchet.sh
- ./scripts/check_fips_wording.sh
- ./scripts/check_embedded_secrets.sh
- make script-tests
- python3 ./scripts/test_object_generation_protocol_model.py
- python3 ./scripts/check_test_wiring.py
- ./scripts/check_no_planning_docs.sh
- ./scripts/check_uring_lane_lib_only.sh

test-count ratchet OK: 52 crates, 22009 tests, 165 ignored match .config/test-count-baseline.txt
PASS: cold stampede script self-test
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
multi-slot round schedules: 640
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
2026-10-16 17:32:35 +08:00
overtrue f6097d1df2 fix(ci): restore integration test compilation
Gate the diagnostic test import with its feature, initialize the conditional
MetadataWriteStore fields explicitly, and decode the scanner replication fixture
through its existing cache boundary. Preserve the original fixture and assertions.

Validation was run on tree 544acb80c0 at 6003d4b8.
LANG=en_US.UTF-8; no source changes follow validation.

Quick checks: 19/19 passed.
- shellcheck --version && actionlint: exit 0
- cargo fmt --all --check: exit 0
- ./scripts/check_unsafe_code_allowances.sh: exit 0
- ./scripts/check_layer_dependencies.sh: exit 0
- ./scripts/check_architecture_migration_rules.sh: exit 0
- ./scripts/check_logging_guardrails.sh: exit 0
- ./scripts/check_error_other_format_ratchet.sh: exit 0
- ./scripts/check_no_tokio_io_uring.sh: exit 0
- ./scripts/check_extension_schema_boundaries.sh: exit 0
- ./scripts/check_body_cache_whitelist.sh: exit 0
- ./scripts/check_s3s_footprint.sh: exit 0
- ./scripts/check_test_count_ratchet.sh: exit 0
- ./scripts/check_fips_wording.sh: exit 0
- ./scripts/check_embedded_secrets.sh: exit 0
- make script-tests: exit 0
- python3 ./scripts/test_object_generation_protocol_model.py: exit 0
- python3 ./scripts/check_test_wiring.py: exit 0
- ./scripts/check_no_planning_docs.sh: exit 0
- ./scripts/check_uring_lane_lib_only.sh: exit 0

Darwin validation:
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets -- -D warnings: exit 0
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries)' --test-threads 1 --no-fail-fast: exit 0, 2/2 passed
- cargo clippy --locked -p rustfs-ecstore -p rustfs-scanner --all-targets --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -- -D warnings: exit 101
- cargo nextest run --locked -p rustfs-ecstore -p rustfs-scanner --lib --features rustfs-ecstore/diagnostic-read-version-caller,rustfs-scanner/diagnostic-read-version-caller -E 'test(config::com::tests::save_config_preserves_retryable_system_volume_errors) | test(scanner_io::tests::replication_maintenance_refreshes_compacted_stats_across_cycles_and_retries) | test(cluster::rpc::runtime_sources::tests::) | test(config::com::tests::diagnostic_config_write_retains_origin_and_api_for_results_and_cancellation)' --test-threads 1 --no-fail-fast: exit 0, 14/14 passed

Diagnostic strict Clippy remains red on four unchanged production lints:
runtime_sources.rs if_same_then_else and three redundant_locals reports in
set_disk/core/metadata_read.rs and set_disk/read.rs. Their source ranges and
feature wiring match the base; the base was not compiled separately.

Linux aarch64 validation (Rust/Clippy 1.99.0, UID 501, jobs=1):
- cargo clippy --all-targets -- -D warnings: exit 0; Finished dev profile in 68m 12s
- cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings: exit 0; Finished dev profile in 49m 45s
Source HEAD/tree and clean status matched before and after both commands.
No compiler or stack overrides were used. protoc 35.1 was installed from its
verified ARM64 release; flatc was absent and not invoked by the workspace build scripts.
The local environment differs from hosted CI.
2026-10-16 13:06:24 +08:00
rustfs-cla-assistant[bot] 7e5d32eecf Merge remote-tracking branch 'origin/main' into gateway/integration 2026-10-16 02:09:53 +00:00
overtrue 72da0e21c4 chore: merge main into gateway/integration
Keep the migrated compatibility adapters in their active modules while bringing in upstream fixes. Preserve native publication history on the new create, rename and borrowed-open paths.

Validation:
- `cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(disk::os::native_creation_history_tests::) | test(disk::local::test::borrowed_open_cancellation_retains_started_history_before_execution) | test(disk::local::test::owned_direct_child_open_attempt_consumes_native_history)' --no-fail-fast`: 9/9 passed after restoring all 12 mutation cases at cea78f2590645eb0d303cd7cf56f4dd0d9c00e87; these source blobs are unchanged in this merge.
- `cargo nextest run --locked -p rustfs-s3select-query --lib -E 'test(dispatcher::json_document_schema::tests::)' --no-fail-fast`: 5/5 passed at 5421ba682d1329cc7ebeaac7c6a863da2cea1996, with the same tree.
- `cargo fmt --all --check`: passed.
- `./scripts/check_test_count_ratchet.sh --update`: 52 crates, 22007 tests, 165 ignored.
- Profile baselines use seven actual Darwin listings and seven actual Linux listings.
- Earlier conflict snapshot f5e0cdf0c58aed1ebe53b9243cf52f94832ab049: `cargo nextest run --locked -p rustfs-ecstore -p rustfs --lib -E 'test(disk::os::) | test(bucket::metadata_sys::) | test(server::legacy_compat::) | test(server::layer::tests::request_logging_watchdog) | test(server::http::tests::conditional_object_http::) | test(storage::helper::tests::operation_helper)' --no-fail-fast` passed 218/218 before the borrowed-open fix. This is historical evidence.
2026-10-16 09:49:08 +08:00
Chris 78a0cf321c refactor(replication): move the replication config onto gateway types (#8443)
* wip: T1.4 replication family

* refactor(replication): move the replication config onto gateway types

The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.

* test: use S3 serializer for list response tests
2026-10-16 06:19:32 +08:00
overtrue cbe38a120f Merge main into gateway/integration
Merge frozen main acd26afd8b and include
compatibility fixes and measured Darwin/Linux membership corrections
across six files.

Validation on candidate b4c3ab560a:
- All 20 local gates passed.
- Native smoke: 218 passed. Native full: 661 passed.
- Full passed on the same-head failed-only rerun; the initial failure
  remains recorded.
- Gateway remains report-only: 38 passed, 180 failed, under the existing
  exemption.

CI: https://github.com/rustfs/rustfs/actions/runs/38285788334
2026-10-13 03:26:16 +08:00
overtrue db0828aa1c test(ecstore): isolate rollback guard loss fixtures
Run the three rollback metadata-cleanup cases in separate test processes so reused /proc/self/fd aliases from earlier cases cannot invalidate their creation witnesses. Keep the original fixture, guards, assertions and io_uring test selection, and verify that each child runs the intended case successfully.

Update the generated test-count baseline for the added macro arm.

Validation: cargo check -p rustfs-ecstore --tests; cargo fmt --all --check; all 19 quick-check commands, including make script-tests and the two Python checks. Linux io_uring acceptance and fault controls remain pending.
2026-10-12 00:42:51 +08:00
overtrue 3ca889b099 fix(ecstore): box nested bucket rollback state
Keep the guarded rollback future behind a heap allocation so bucket creation does not store its nested state inline. The rollback body, all three fence references and pending-proof revocation order are unchanged.

The same three original Linux tests overflowed on both e611 CI attempts. Local format, ecstore test compilation and all 19 quick-check steps pass; the original Linux feature lanes still need to validate this candidate.
2026-10-11 20:32:00 +08:00
overtrue e6112c526f fix(ecstore): retain creation fences through rollback
Keep lifecycle, metadata-transaction, and namespace guards around rollback and metadata cleanup. Preserve the original publication error when a guard is lost. Build the timestamp retry fixture through the real interrupted producer so it checks a valid pending creation.

Validated source tree: 4e8e207999.
Two independent source reviews found no blocking regression.

Validation (LANG=en_US.UTF-8):
- bash -c 'shellcheck --version && actionlint' — exit 0
- cargo fmt --all --check — exit 0
- ./scripts/check_unsafe_code_allowances.sh — exit 0
- ./scripts/check_layer_dependencies.sh — exit 0
- ./scripts/check_architecture_migration_rules.sh — exit 0
- ./scripts/check_logging_guardrails.sh — exit 0
- ./scripts/check_error_other_format_ratchet.sh — exit 0
- ./scripts/check_no_tokio_io_uring.sh — exit 0
- ./scripts/check_extension_schema_boundaries.sh — exit 0
- ./scripts/check_body_cache_whitelist.sh — exit 0
- ./scripts/check_s3s_footprint.sh — exit 0
- ./scripts/check_test_count_ratchet.sh — exit 0
- ./scripts/check_fips_wording.sh — exit 0
- ./scripts/check_embedded_secrets.sh — exit 0
- make script-tests — exit 0
- python3 ./scripts/test_object_generation_protocol_model.py — exit 0
- python3 ./scripts/check_test_wiring.py — exit 0
- ./scripts/check_no_planning_docs.sh — exit 0
- ./scripts/check_uring_lane_lib_only.sh — exit 0
- cargo nextest run --profile ci -p rustfs-ecstore --lib --locked --no-fail-fast — 5894 passed, 0 skipped
- cargo check -p rustfs-ecstore --all-targets --locked — exit 0
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(=admin::handlers::table_catalog::tests::cold_strong_registration_completes_while_other_bucket_backup_waits) | test(=admin::handlers::table_catalog::tests::commit_backend_borrowed_put_preserves_owner_authorization_and_precondition) | test(=admin::handlers::table_catalog::tests::disable_table_bucket_preserves_objects_and_can_be_reenabled) | test(=admin::handlers::table_catalog::tests::load_table_handler_negotiates_vended_credentials_without_breaking_metadata_only_callers) | test(=admin::handlers::table_catalog::tests::namespace_property_handler_updates_object_backed_catalog_and_maps_errors) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body) | test(=table_catalog::store::tests::catalog_writer_loss_prevents_recovery_storage_commit)' — 7 passed
- cargo nextest run --profile ci -p rustfs --lib --locked --no-fail-fast -E 'test(unsupported_remote_disk_)' — 2 passed

Additional fault-injection evidence remains incomplete. Timestamp M1 reached its original assertion; its restore run is being collected. M2–M5 and the three planned rollback faults have not run. An automated risk check interrupted the timestamp proof task; no assertions or validation settings were weakened.
2026-10-11 16:16:31 +08:00
Chris c5967686f1 fix(ecstore): bound bucket creation stack usage (#8466) 2026-10-11 00:55:29 -07:00
Chris 859de565ad refactor(rpc): serve internode rpc bodies with the owned Body (#8441)
* test(rpc): pin the wire bytes of every internode rpc response body

rustfs/backlog#2747 batch 4, step 1 of 2, on the s3s bodies the rpc paths
serve today. Each response body builder in storage/rpc/http_service.rs is
served through hyper's HTTP/1.1 server and the raw bytes the peer reads are
compared with goldens captured from this commit: status, content-length or
chunked framing, chunk boundaries, the terminating chunk, HEAD, and the cut
connection a failure after data produces. The capability responses carry a
per-process proof, so their length header and payload are checked against
the body they encode instead.

The read_file_stream response builder moves into read_file_stream_response
so the test reaches the same code the handler runs; behaviour is unchanged.

* refactor(rpc): serve internode rpc bodies with the owned Body

rustfs/backlog#2747 batch 4, step 2 of 2. The internode rpc HTTP paths
build their responses with rustfs_s3_types::Body: Body::from_stream
replaces Body::from(StreamingBlob::wrap(..)) at the six stream sites, and
the in-memory and empty bodies keep their constructors. InternodeRpcService
converts its responses into the wrapped service's response body through
From<Body>, so the server keeps composing it with the s3s service as
before. The wire-byte goldens from the previous commit pass unchanged;
storage/rpc no longer names s3s.

* fix(rpc): simplify wire test response builder type

Name the boxed response builder used by the capability wire test so
all-target Clippy can check it without a type-complexity warning.

Validation (LANG=en_US.UTF-8, CARGO_INCREMENTAL=0):
- cargo clippy -p rustfs --all-targets -- -D warnings: exit 0.
- cargo nextest run --profile ci -p rustfs --lib -E 'test(/storage::rpc::/)': 279 tests run, 279 passed, 4842 skipped.
- All 19 quick-check commands, including script-tests, both Python scripts and cargo fmt --all --check: exit 0.
- Footprint baseline regenerated: files=194, s3_error_lines=1584, ecstore_files=36; test-count baseline regenerated: 52 crates, 21228 tests, 164 ignored.
- All 54 Cargo.toml files parse without duplicate keys.

No assertion was added or changed by this lint fix. R1-R8 are retained
in the existing PR verification record.

* fix(ecstore): resume fenced Object Lock creation intents

* fix(ecstore): bound creation retry futures on the stack

* fix(ecstore): require native proof for creation retries
2026-10-11 12:34:14 +08:00
overtrue 1f7cc593e7 test(ecstore): isolate tier-free recovery fixture startup
The fixture constructor can take the production recovery lock and wait for 60 seconds before this test installs its queue and seeds metadata. Disable its background recovery only while creating this fixture, then restore the environment before calling the existing production recovery entrypoint.

Only the existing test setup changes. Its production entrypoint, default lock parameters, original assertions, 30-second timeout, cancellation and cleanup remain intact. No tests or assertions were added. Production and store/init.rs are byte-identical to the parent.

Independent acceptance ran on merge tree 8ec358cb42ba9ebb3bcfeafcef048a3f0366355b; the runner header reports parent HEAD, while before/after index and source hashes confirm this candidate. Worker task branch d8055766b8 has the same tree.

Every command used LANG=en_US.UTF-8. Task compilation also used CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, CARGO_NET_OFFLINE=true, an independent target, and empty RUSTC_WRAPPER/RUSTC_WORKSPACE_WRAPPER to keep compilers in its owned process group without changing repository configuration.

Command:
shellcheck --version && actionlint
Exit 0; observed output excerpt:
version: 0.11.0
Log SHA-256: fc9eec16b4191d058a295813ca9afbe9d2b37b59975da860170726be162a3924

Command:
cargo fmt --all --check
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_unsafe_code_allowances.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_layer_dependencies.sh
Exit 0; observed output excerpt:
Layer dependency guard passed (no new reverse dependencies/cycles).
Log SHA-256: 69c6cb1896c0586b28d1167380d3260d99d8017a4265c8df5a2348f289e93a12

Command:
./scripts/check_architecture_migration_rules.sh
Exit 0; observed output excerpt:
Architecture migration rules passed.
Log SHA-256: 28790e70d070607ed4de0704f292a65030be92ec224ef106d4b72fd8296f896e

Command:
./scripts/check_logging_guardrails.sh
Exit 0; observed output excerpt:
✅ Logging guardrails check passed
Log SHA-256: 7adf41abff8ed8f7617decf86498888ddde78937099dd9ce47f6d359dfc2d317

Command:
./scripts/check_error_other_format_ratchet.sh
Exit 0; observed output excerpt:
error(format!) ratchet passed (total call sites: 579).
Log SHA-256: 66172aef18f150ba0605af122e26ffcf91f6dd86f43e38b05e2e982ce8d783ee

Command:
./scripts/check_no_tokio_io_uring.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_extension_schema_boundaries.sh
Exit 0; observed output excerpt:
Extension schema boundaries passed.
Log SHA-256: f13d0acd0e2ce5d9815241b12b4dab7d01cc11b9b65bcf6dfb7812c878532a51

Command:
./scripts/check_body_cache_whitelist.sh
Exit 0; observed output excerpt:
Body-cache whitelist guard passed.
Log SHA-256: 6b828847b0383950ee6eef9cd846d0c322c945e47efd4c7d1ad881178810d6bb

Command:
./scripts/check_s3s_footprint.sh
Exit 0; observed output excerpt:
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
Log SHA-256: 47f53d04ef645618563fe1f22abc64667f19078f1974feedcca46e4f2c016844

Command:
./scripts/check_test_count_ratchet.sh
Exit 0; observed output excerpt:
test-count ratchet OK: 52 crates, 21340 tests, 164 ignored match .config/test-count-baseline.txt
Log SHA-256: 618a5e330e933de0c6ff484546df401457024a88bb5986d11d588794e33dab60

Command:
./scripts/check_fips_wording.sh
Exit 0; observed output excerpt:
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
Log SHA-256: 99221cc9979c480ec4f46f50d76c52b18c72fe499f12e15e4f4a22a1b8bfb140

Command:
./scripts/check_embedded_secrets.sh
Exit 0; observed output excerpt:
Embedded secret guard passed (no private key material or provider credential literal in the tree).
Log SHA-256: 437a3ceba32094d2901889965ad191e411aad32cdb18bf0b8659e818011c406b

Command:
make script-tests
Exit 0; observed output excerpt:
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok
Ran 10 tests in 0.042s
OK
Ran 30 tests in 20.627s
OK
Ran 10 tests in 2.268s
OK
Ran 33 tests in 0.127s
OK
Ran 7 tests in 0.002s
OK
Ran 7 tests in 0.001s
OK
Ran 4 tests in 0.108s
OK
Ran 3 tests in 0.564s
OK
PASS: cold stampede script self-test
Log SHA-256: 75fda2d069a373c63667b908baaf99e6591aab9099a739b0c4c22ceb6da8d68c

Command:
python3 ./scripts/test_object_generation_protocol_model.py
Exit 0; observed output excerpt:
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok
Ran 24 tests in 0.929s
OK
Log SHA-256: 0b4be926efb0609d497ae58930340153f01957b606206cbbdb3e0d47eb29d3a2

Command:
python3 ./scripts/check_test_wiring.py
Exit 0; observed output excerpt:
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
Log SHA-256: 21f676f458451525535ab0ef73d6a9c0d9774e16d76a4d19e396ad34936dd39c

Command:
./scripts/check_no_planning_docs.sh
Exit 0; observed output excerpt:
(no stdout or stderr)
Log SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Command:
./scripts/check_uring_lane_lib_only.sh
Exit 0; observed output excerpt:
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
Log SHA-256: 43f3f851e27ec2ab9e34d6b05fefdede38af992699c473904bfe6f53a46d52e3

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast
Exit 0; wall 234.399s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3m 49s
    Starting 1 test across 1 binary (5871 tests skipped)
  SETUP PASS [   0.006s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.440s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item
     Summary [   0.586s] 1 test run: 1 passed, 5871 skipped
Log SHA-256: 34e973f86e8dc2102c1a0e483e0ac802ce7df3d1b40a603932f62c0805c4de27

Command:
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over) | test(=store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence) | test(=store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable)' --no-fail-fast
Exit 0; wall 4.010s.
Observed output:
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.43s
    Starting 3 tests across 1 binary (5869 tests skipped)
  SETUP PASS [   0.005s] ecstore-base-stack: sh -c 'echo RUST_MIN_STACK=4194304 >> "$NEXTEST_ENV"'
        PASS [   0.068s] rustfs-ecstore bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_eight_coordinators_hold_one_leader_and_take_over
        PASS [   0.637s] rustfs-ecstore store::bucket::tests::creation_commit_backfill_runs_under_the_config_write_fence
        PASS [   0.846s] rustfs-ecstore store::bucket::tests::orphaned_bucket_recovery_requires_every_volume_absent_and_is_retryable
     Summary [   0.987s] 3 tests run: 3 passed, 5869 skipped
Log SHA-256: b5da2a927ff18ca9aa217762f7186a6d4371bd9e8feedd7b77940e10b64dffac

Command:
cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Exit 0; wall 1084.508s.
Observed output:
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 18m 03s
Log SHA-256: 28f98be7970faaa71cd118da1da4e6e2e3d23b9dda7e6449eaa7a37e741260b5

Compiled mutation command (unchanged for each red and restored run):
cargo nextest run --profile ci -p rustfs-ecstore --lib --features rio-v2 -E 'test(=bucket::lifecycle::bucket_lifecycle_ops::tests::tier_free_version_recovery_production_entrypoint_enqueues_seeded_item)' --no-fail-fast

M1: reverse only this test-local constructor flag from false to true. The mutant compiled, then failed the original recovered-item queue assertion after 30.481s. Restoring the entire observed candidate passed in 0.460s. Red log 55bf3b1535529ac0bcfb8071f9b56f9f40631007a85c315367e7e93786d65293; restored log 204d5fbe4574a7917f5e740a795d864fe28a7e7fb5e33d27b0a9cf68e2f17f52.

M2: replace only the actual awaited enqueue with false, consuming its original inputs. The mutant compiled, then failed the same original queue assertion after 30.516s. Restoring the entire observed candidate passed in 0.457s. Red log 89186330413215e231e2b06bf013cc31097ea8f4a106d91421a36d6c8986a7fd; restored log e848b42d2302401afa5dbcd9e37bf76d2e56a96ab6de5a488513b997f53c5e9b.

All temporary observation and mutation code was removed before final acceptance. Historical baseline and diagnostic failures remain recorded; they are not mutation credit. The separate Object Lock intent-retry diagnostic still fails, and these native macOS checks do not establish Linux CI recovery.
2026-10-09 19:29:50 +08:00
overtrue 1d5a50f094 fix: remove redundant pending closure in disk test
Pass the capture-free pending future factory as a function item in the existing slow-operation test. This fixes the redundant_closure error at disk_store.rs:2914 that stopped strict Clippy before the feature tests could run.

The single-line test edit preserves the factory's result, warning and timeout behavior. Reversing the replacement reproduces the parent file byte for byte. All 220 assertion lines and the production prefix are unchanged; no tests, assertions, guards, profiles or baselines were added or relaxed.

Original Linux CI evidence (run 37888552268 at 53245be9f):
- RIO job113685483992: strict Clippy exit101, redundant closure at disk_store.rs:2914; feature tests skipped.
- Workspace Test and Lint job113685484058: same strict Clippy exit101; nextest and doc tests skipped.
These are lint failures, not runtime mutation kills or known flakes.

Native macOS arm64 validation on tree7a5bf47e7533b29479bc85207416102f1337c566, LANG=en_US.UTF-8, one build job, CARGO_INCREMENTAL=0 and offline dependencies:

shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
All 19 checks: exit0, 312.161s.

$ cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 20m 11s
exit0, 1213.210s total.

$ cargo nextest run --profile ci -p rustfs-ecstore --features rio-v2 slow_disk_operation_warns_once_while_pending_and_preserves_result --no-fail-fast
Finished `test` profile [unoptimized + debuginfo] target(s) in 4m 34s
PASS [0.026s] rustfs-ecstore disk::disk_store::tests::slow_disk_operation_warns_once_while_pending_and_preserves_result
Summary [   0.170s] 1 test run: 1 passed, 5943 skipped
exit0, 281.920s total; 5943 existing filter skips.

warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
This linker warning is retained; flags, profiles and allowances are unchanged.

54 tracked Cargo.toml files parsed without duplicate keys. git diff --check passed. No new assertions were introduced, so no new runtime mutation is claimed. Native compilation and the existing helper test do not substitute for actual Linux RIO execution; the new integration CI run still needs to recover before the RPC merge.

Preservation and output hashes:
parent file SHA256: 8e59ee745e3aaaa22848b943d3b10f6353b515956282cde1f3af2c394200c9ac
final file SHA256: b69ba3237061b161448ae8b62b615885d5770b98a5b4a2479bcfbd70f1bd1f17
unchanged production prefix SHA256: 7903a6e288252c420231242d7b58a25fa2a7140ea3dd430cbe7f0dde11369dcb
RIO Linux failure log SHA256:421addb9868fd88e4109a922311e54f91f0dfd70862f8ba7fcaba193482ed90b
Workspace Linux failure log SHA256:e0aea8d0f014c022252a9534968e083c98cfc1fd760da445ecd04ec3f522e26a
19 output SHA256:35a54f28d707956ed1edb29d492c4dfe8a9fe753d103b111b5a2f1b5c41c2b98
RIO Clippy output SHA256:d6046e3868f2b3992fe11cdd04e9e98eb50a87d1234d9a70dbfeb154ff61638d
slow-operation output SHA256:605b7b258d2fd09c6ef2ab2ffc90d433d0509480f1e45125d20c823759e83608

Refs rustfs/backlog#2734.
2026-10-09 14:44:39 +08:00
overtrue 53245be9f4 fix: repair integration checks and IPC test readiness
Wait for the RPC/locks test sockets to pass the existing owner and mode
checks inside the original 5s readiness window and 10ms poll interval.

Use the existing admin S3 carrier for the new recovery and quota error
calls. Regenerate the test-count baseline for main's 118 existing tests
across seven crates. Pair the private pagination helper's raw-scan inputs
and remove the redundant Copy clone from the retirement event.

Keep the original assertions, responses, authorization, locks and storage
operations. The pagination body and raw-scan expressions at all five calls
are byte-identical; reinserting only the Copy clone restores the entire
base heal file. All ignored baseline values and s3s counts stay unchanged.

Validation base: c47432e353
Validation tree: f081084bc5
Patch SHA-256: 2dcf0028691bfb2772edfb38e262829b04f4151832dcce823c9dd9f7434b52de
Trace production-prefix SHA-256: fd994f218a21429d62866f4efb8eee1ee57b58f4a51865aa35d05a54ef1ccb45

All commands used LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1,
CARGO_INCREMENTAL=0 and CARGO_NET_OFFLINE=true in one exclusive target.
Every gate retained the same tree and six source hashes before and after.
54 Cargo.toml files parse without duplicate keys.

cargo clippy -p rustfs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 12m 50s
Exit: 0. Wall time: 770.855s.

The original 19 quick-check steps:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0
Wall time: 286.337s.

cargo nextest run --profile ci -p rustfs --no-fail-fast
     Summary [ 217.017s] 5382 tests run: 5382 passed (4 leaky), 40 skipped
Exit: 0. Wall time: 1324.414s.

cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
     Summary [   2.189s] 543 tests run: 543 passed, 0 skipped
Exit: 0. Wall time: 123.192s.

cargo nextest run --profile ci -p rustfs-ecstore list_objects_paginate --no-fail-fast
     Summary [   0.170s] 8 tests run: 8 passed, 5921 skipped
Exit: 0. Wall time: 261.324s.

cargo nextest run --profile ci -p rustfs-heal durable_mrf_anchor_retires_only_after_exact_bucket_generation_is_deleted --no-fail-fast
     Summary [   0.805s] 1 test run: 1 passed, 835 skipped
Exit: 0. Wall time: 199.801s.

Baseline regeneration:
./scripts/check_test_count_ratchet.sh --update
wrote .config/test-count-baseline.txt: 52 crates, 21340 tests, 164 ignored
Test-only increases: ecstore48, heal3, iam14, io-metrics2, obs3, policy14,
rustfs34. Existing ignored values and all other rows are byte-identical.
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)

Historical runtime red controls on f622 (same retained two assertions):
cargo nextest run --profile ci -p rustfs --lib -E 'test(=connect::diagnostics::trace_runtime::tests::local_top_rpc_reads_service_http_completions_and_signs_offline) | test(=connect::diagnostics::trace_runtime::tests::local_top_locks_reads_service_manager_and_signs_offline)' --no-fail-fast
Original path readiness with a temporary 500ms bind/seal gap:
     Summary [   0.863s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Sealed readiness in that same gap:
     Summary [   5.621s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Compiled M1: replace the helper predicate with path.exists():
     Summary [   0.870s] 2 tests run: 0 passed, 2 failed, 5113 skipped
Exit: 100.
Restore the sealed predicate byte-for-byte:
     Summary [   5.640s] 2 tests run: 2 passed, 5113 skipped
Exit: 0.
Both reds observe mode755 with owner_matches=true and fail at the original
RPC/locks unwraps with StateSecurity. M1 failure locations: trace_runtime.rs
2547:29 and 2713:29. No new assertions were introduced.

cargo nextest run --profile ci -p rustfs --lib -E 'test(connect::diagnostics::trace_runtime::tests::)' --no-fail-fast
     Summary [  20.748s] 28 tests run: 28 passed (1 leaky), 5087 skipped
Exit: 0. This security sweep is historical; the full final unit gate above
covers the actual c474 candidate.
The first disk-stopped compile ran no tests and is excluded from red credit.
All temporary seal instrumentation is removed from the final source.

Final unit cleanup markers: 4 leaky, 40 original skips (exit0). The ld
__eh_frame linker warning also appears verbatim in the original f622 run
and the earlier c474 run; there are no new source warnings.
2026-10-09 13:09:50 +08:00
rustfs-cla-assistant[bot] c47432e353 Merge remote-tracking branch 'origin/main' into gateway/integration 2026-10-09 02:20:19 +00:00
overtrue f622c1bce0 fix(test): compare legacy body wire parity as raw bytes
Lossy UTF-8 comparison concealed a changed binary response byte.
Compare the already captured Vec<u8> values directly.

Validation:
- Binary mutant: migrated payload 0x84 vs legacy 0x83; old comparator
  passed 1/1, raw comparator failed 0/1 (exit 100); restored green.
- Non-Swift body-drain mutant: untouched-body delegation test failed
  with empty vs payload (exit 100); restored handler without a diff.
- cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols
  --features rustfs-protocols/swift --no-fail-fast:
  543 tests run, 543 passed, 0 skipped after both mutants restored.
- cargo fmt --all --check and git diff --check passed.

Orchestrator acceptance (LANG=en_US.UTF-8):
cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
Summary [2.445s] 543 tests run: 543 passed, 0 skipped.

All shared quick checks on the final code tree:
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

Final independent acceptance after the drive-test CI repair
Parent: a99d8f4f89
Tested code tree: 93cbd96bf0

Tested tree: 075ae5161c2b62a96812bc1e3c904a7663b4f2f8
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs-s3-types -p rustfs-protocols --features rustfs-protocols/swift --no-fail-fast
    Finished `test` profile [unoptimized + debuginfo] target(s) in 10m 08s
 Nextest run ID 5c7ff102-ec6d-479a-86e0-810859b28bbf with nextest profile: ci
    Starting 543 tests across 8 binaries
     Summary [   6.729s] 543 tests run: 543 passed, 0 skipped
exit_code=0

The only source change is the raw-byte wire comparator. Cargo manifests and the ratchet baselines remain unchanged from the independently validated parent. The invalid-byte and consumed-request mutations remain recorded above; their implementation changes were restored.
2026-10-09 08:42:46 +08:00
overtrue a99d8f4f89 test(connect): restore one-second drive request budgets
Apply the exact test-only repair already landed on main in 5972a09cc3. The filesystem probe and production CLI success test request one second and keep their existing one-second duration and parameter checks. All outcome, I/O, error, cleanup and resource-limit assertions remain.

Scope: rustfs/tests/connect_perf_drive.rs only. Patch-id: cb0a1dfc8f68a67727525adea7458e2cee7d09f8.

Validation environment: LANG=en_US.UTF-8, CARGO_BUILD_JOBS=1, CARGO_INCREMENTAL=0, exclusive APFS-cloned CARGO_TARGET_DIR.

Quick Checks: every command from .github/actions/quick-checks/action.yml, with captured output and exit status below.

$ shellcheck --version && actionlint
ShellCheck - shell script analysis tool
version: 0.11.0
license: GNU General Public License, version 3
website: https://www.shellcheck.net
exit_code=0

$ cargo fmt --all --check
exit_code=0

$ ./scripts/check_unsafe_code_allowances.sh
exit_code=0

$ ./scripts/check_layer_dependencies.sh
Layer dependency guard passed (no new reverse dependencies/cycles).
exit_code=0

$ ./scripts/check_architecture_migration_rules.sh
Architecture migration rules passed.
exit_code=0

$ ./scripts/check_logging_guardrails.sh
✅ Logging guardrails check passed
exit_code=0

$ ./scripts/check_error_other_format_ratchet.sh
error(format!) ratchet passed (total call sites: 582).
exit_code=0

$ ./scripts/check_no_tokio_io_uring.sh
exit_code=0

$ ./scripts/check_extension_schema_boundaries.sh
Extension schema boundaries passed.
exit_code=0

$ ./scripts/check_body_cache_whitelist.sh
Body-cache whitelist guard passed.
exit_code=0

$ ./scripts/check_s3s_footprint.sh
s3s footprint OK: files importing s3s is 195 (baseline files=195)
s3s footprint OK: s3s s3_error! lines is 1584 (baseline s3_error_lines=1584)
s3s footprint OK: ecstore files referencing s3s is 36 (baseline ecstore_files=36)
✅ s3s footprint baseline check passed
exit_code=0

$ ./scripts/check_test_count_ratchet.sh
test-count ratchet OK: 52 crates, 21222 tests, 164 ignored match .config/test-count-baseline.txt
exit_code=0

$ ./scripts/check_fips_wording.sh
FIPS wording guard passed (README.md and CHANGELOG.md contain no forbidden claims).
KMS wording guard passed (crates/kms claims no Vault KV2 Transit wrapping).
exit_code=0

$ ./scripts/check_embedded_secrets.sh
Embedded secret guard passed (no private key material or provider credential literal in the tree).
exit_code=0

$ make script-tests
Running script tests...
entrypoint credential policy tests passed
internode grpc A/B bench env tests passed
object batch benchmark enhanced tests passed
PASS package service script contract
hotpath warp A/B gate tests passed
hotpath warp ABBA tests passed
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out
Scanner validation artifacts written to /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.1pFe6Kl1Yu/out-missing-pid
.............................
----------------------------------------------------------------------
Ran 29 tests in 181.286s

OK
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G01","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G02"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-E"}
{"decision":"verified","pending_gates":["G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G01"}
{"decision":"verified","pending_gates":["G01","G02","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G03"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-L"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G14"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G10"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P1"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P2"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P3"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
Status-and-outcome release descriptor verified: /private/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp8rwn1pe0/descriptor/release-bundle-status-outcome.json
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G05"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G06"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"R-D"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G07"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G09","G10","G11","G12","G13","G14","P1","P2","P3","P4","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"G08"}
{"decision":"verified","pending_gates":["G01","G02","G03","G04","G05","G06","G07","G08","G09","G10","G11","G12","G13","G14","P1","P2","P3","R-D","R-E","R-L"],"pending_lanes":["authority-coverage","checkpoint-and-crash","ec8-4-multiset","maintenance-producers","mixed-version-rollback","mrf-responsibility","scheduler-pressure","status-and-outcome"],"rejected_errors":[],"rejected_gate":null,"release_approved":false,"release_schema_capable":true,"schema":1,"verified_gate":"P4"}
exact 1MiB handoff ABBA tests passed
pinned paired ABBA bench tests passed
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=7, -seed=123456789, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/explicit-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=123456789
target=path_containment
seed=123456789
max_total_time=7
git_revision=4d15b816236630b2c42f321ffc9ed3654edc94e3
git_dirty=true
runner_mode=prebuilt
==> /var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/bin/path_containment (-max_total_time=1, -seed=2322617706, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/auto-artifacts/path_containment/, corpus=/private/tmp/gw-orchestrator-2734/ci-drive/fuzz/corpus/path_containment)
-seed=2322617706
==> cargo +nightly fuzz run path_containment (-max_total_time=9, -seed=987654321, -artifact_prefix=/var/folders/0w/s4wxsn2n4w9b6f3cd4xrz53w0000gn/T/tmp.mYmuI01hNq/cargo-artifacts/path_containment/)
+nightly
fuzz
run
path_containment
-seed=987654321
seed=987654321
runner_mode=cargo-fuzz
fuzz runner seed manifest tests passed
✅ scripts/python_bin.sh resolver checks passed
self-test ok: clean fixture with every exemption present passes
self-test ok: all 11 pattern families fire
self-test ok: moving a pinned source is reported
Embedded secret guard self-test passed.
self-test ok: p1-s3s-import: 'use s3s::s3_error;' counts every line, the rustfs_s3_types module's too (s3_error_lines=3)
self-test ok: p2-s3s-mixed-tree: a multi-line s3s use tree naming s3_error counts (s3_error_lines=3)
self-test ok: p3-s3s-glob: a crate-root glob of s3s, bare or inside a tree, counts (s3_error_lines=5)
self-test ok: p4-macro-use: '#[macro_use] extern crate s3s' counts (s3_error_lines=3)
self-test ok: p5-qualified-s3s: a qualified s3s::s3_error! line counts in a cleared file (s3_error_lines=2)
self-test ok: p6-exact-pass: a baseline equal to the tree passes (exit 0)
self-test ok: n1-rustfs-import: 'use rustfs_s3_types::s3_error;' alone counts nothing (s3_error_lines=1)
self-test ok: n2-rustfs-mixed-tree: a rustfs_s3_types tree plus an s3s tree without the macro counts nothing (s3_error_lines=1)
self-test ok: n3-defines: the file defining the macro counts nothing (s3_error_lines=1)
self-test ok: n4-qualified-rustfs: a qualified rustfs_s3_types::s3_error! line counts nothing (s3_error_lines=1)
self-test ok: n5-renamed-call: a call through a renamed macro is not text-matched (out of scope) (s3_error_lines=1)
self-test ok: n6-comment-string: a comment or string mentioning s3_error! in a cleared file counts nothing (s3_error_lines=1)
self-test ok: n7-e2e-excluded: crates/e2e_test is excluded (s3_error_lines=1)
self-test ok: n8-super-glob: an unrecognised import form ('use super::*;') counts (s3_error_lines=3)
self-test ok: n9-facade: a crate-local re-export counts (s3_error_lines=2)
self-test ok: n10-renamed-import: a renamed rustfs_s3_types import does not clear s3_error! lines (s3_error_lines=2)
self-test ok: n11-commented-import: a commented-out rustfs_s3_types import does not clear (s3_error_lines=2)
self-test ok: n12-rustfs-glob: a glob import of rustfs_s3_types does not clear (s3_error_lines=2)
self-test ok: n13-comment-only: a comment mentioning s3_error! in a file with no import counts (s3_error_lines=2)
self-test ok: f1-baseline-missing: a missing baseline file fails (exit 1)
self-test ok: f2-key-missing: a baseline without s3_error_lines fails (exit 1)
self-test ok: f3-rg-fails: a failing classification search fails the guard (exit 1)
✅ s3s footprint self-test passed
test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs (__main__.SelfTests.test_core_gate_rejects_missing_ignored_filtered_and_corrupt_inputs) ... ok
test_e2e_requires_registration (__main__.SelfTests.test_e2e_requires_registration) ... ok
test_fuzz_runtime_uses_matrix_target (__main__.SelfTests.test_fuzz_runtime_uses_matrix_target) ... ok
test_ilm_lane_keeps_the_measured_cargo_build_budget (__main__.SelfTests.test_ilm_lane_keeps_the_measured_cargo_build_budget) ... ok
test_profile_listing_binds_platform_digest (__main__.SelfTests.test_profile_listing_binds_platform_digest) ... ok
test_profile_listing_enforces_selection (__main__.SelfTests.test_profile_listing_enforces_selection) ... ok
test_quick_checks_commands_propagate_failure (__main__.SelfTests.test_quick_checks_commands_propagate_failure) ... ok
test_quick_checks_rejects_caller_and_execution_bypasses (__main__.SelfTests.test_quick_checks_rejects_caller_and_execution_bypasses) ... ok
test_runner_readiness_fails_closed (__main__.SelfTests.test_runner_readiness_fails_closed) ... ok
test_s3_runner_rejects_unbounded_failure_locals (__main__.SelfTests.test_s3_runner_rejects_unbounded_failure_locals) ... ok
test_scanner_heal_begin_requires_embedded_source_provenance (__main__.SelfTests.test_scanner_heal_begin_requires_embedded_source_provenance) ... ok
test_scanner_heal_case_does_not_approve_pending_release (__main__.SelfTests.test_scanner_heal_case_does_not_approve_pending_release) ... ok
test_scanner_heal_case_lane_is_required_for_runner_profile (__main__.SelfTests.test_scanner_heal_case_lane_is_required_for_runner_profile) ... ok
test_scanner_heal_case_only_schema_cannot_approve_release (__main__.SelfTests.test_scanner_heal_case_only_schema_cannot_approve_release) ... ok
test_scanner_heal_crash_case_rejects_restart_oracle (__main__.SelfTests.test_scanner_heal_crash_case_rejects_restart_oracle) ... ok
test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry (__main__.SelfTests.test_scanner_heal_ec84_case_rejects_wrong_erasure_geometry) ... ok
test_scanner_heal_finish_collects_oracles_from_registry (__main__.SelfTests.test_scanner_heal_finish_collects_oracles_from_registry) ... ok
test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write (__main__.SelfTests.test_scanner_heal_multipool_case_accepts_deferred_optional_outage_write) ... ok
test_scanner_heal_pending_gate_cannot_map_to_implemented_lane (__main__.SelfTests.test_scanner_heal_pending_gate_cannot_map_to_implemented_lane) ... ok
test_scanner_heal_receipts_reject_reuse_and_missing_builds (__main__.SelfTests.test_scanner_heal_receipts_reject_reuse_and_missing_builds) ... ok
test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers (__main__.SelfTests.test_scanner_heal_rejects_boolean_fractional_and_out_of_geometry_integers) ... ok
test_scanner_heal_rejects_broken_execution_and_artifacts (__main__.SelfTests.test_scanner_heal_rejects_broken_execution_and_artifacts) ... ok
test_scanner_heal_rejects_copied_junit_and_wrong_suite_build (__main__.SelfTests.test_scanner_heal_rejects_copied_junit_and_wrong_suite_build) ... ok
test_scanner_heal_release_bundle_accepts_complete_measured_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_accepts_complete_measured_evidence) ... ok
test_scanner_heal_release_bundle_assembler_copies_measured_artifacts (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_copies_measured_artifacts) ... ok
test_scanner_heal_release_bundle_assembler_merges_measured_descriptors (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_merges_measured_descriptors) ... ok
test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_duplicate_descriptor_gate) ... ok
test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_fixture_descriptor) ... ok
test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_hash_mismatch) ... ok
test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions (__main__.SelfTests.test_scanner_heal_release_bundle_assembler_rejects_mixed_descriptor_revisions) ... ok
test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions (__main__.SelfTests.test_scanner_heal_release_bundle_enforces_topology_duration_profile_and_versions) ... ok
test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_fixture_writer_is_not_release_evidence) ... ok
test_scanner_heal_release_bundle_gate_accepts_single_measured_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_accepts_single_measured_gate) ... ok
test_scanner_heal_release_bundle_gate_rejects_missing_field (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_missing_field) ... ok
test_scanner_heal_release_bundle_gate_rejects_unknown_gate (__main__.SelfTests.test_scanner_heal_release_bundle_gate_rejects_unknown_gate) ... ok
test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields (__main__.SelfTests.test_scanner_heal_release_bundle_rejects_synthetic_or_missing_fields) ... ok
test_scanner_heal_release_bundle_requires_domain_evidence (__main__.SelfTests.test_scanner_heal_release_bundle_requires_domain_evidence) ... ok
test_scanner_heal_release_bundle_requires_field_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_field_provenance) ... ok
test_scanner_heal_release_bundle_requires_json_artifact_provenance (__main__.SelfTests.test_scanner_heal_release_bundle_requires_json_artifact_provenance) ... ok
test_scanner_heal_release_bundle_requires_mixed_version_field_roles (__main__.SelfTests.test_scanner_heal_release_bundle_requires_mixed_version_field_roles) ... ok
test_scanner_heal_release_bundle_requires_same_gate_measurement_window (__main__.SelfTests.test_scanner_heal_release_bundle_requires_same_gate_measurement_window) ... ok
test_scanner_heal_release_consumes_multiple_registry_oracles (__main__.SelfTests.test_scanner_heal_release_consumes_multiple_registry_oracles) ... ok
test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence (__main__.SelfTests.test_scanner_heal_release_matrix_lanes_remain_pending_without_real_evidence) ... ok
test_scanner_heal_release_requirements_cannot_be_cleared (__main__.SelfTests.test_scanner_heal_release_requirements_cannot_be_cleared) ... ok
test_scanner_heal_release_status_rejects_focused_case_run (__main__.SelfTests.test_scanner_heal_release_status_rejects_focused_case_run) ... ok
test_scanner_heal_release_status_rejects_synthetic_case (__main__.SelfTests.test_scanner_heal_release_status_rejects_synthetic_case) ... ok
test_scanner_heal_required_evidence_fields_cannot_be_removed (__main__.SelfTests.test_scanner_heal_required_evidence_fields_cannot_be_removed) ... ok
test_scheduled_alerts_require_completion_watchdog (__main__.SelfTests.test_scheduled_alerts_require_completion_watchdog) ... ok
test_update_profile_selection_changes_only_requested_platform (__main__.SelfTests.test_update_profile_selection_changes_only_requested_platform) ... ok
test_validate_includes_quick_checks (__main__.SelfTests.test_validate_includes_quick_checks) ... ok
test_vault_tests_require_cross_process_serialization (__main__.SelfTests.test_vault_tests_require_cross_process_serialization) ... ok
test_workflow_readiness_requires_dependency_probe (__main__.SelfTests.test_workflow_readiness_requires_dependency_probe) ... ok

----------------------------------------------------------------------
Ran 52 tests in 14.301s

OK
e2e-full selection OK: 1 tests, sha256=29e30e6ed2ed38d9e5b0a7cdd9536abf5c719f7021a02b578ae1381563a3c273
...................................
----------------------------------------------------------------------
Ran 35 tests in 1.161s

OK
...................
----------------------------------------------------------------------
Ran 19 tests in 17.397s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 0.183s

OK
test_actual_selector_bootstrap_uses_base_policy_and_fails_closed (__main__.SelfTests.test_actual_selector_bootstrap_uses_base_policy_and_fails_closed) ... ok
test_documentation_paths_do_not_hide_build_or_fixture_changes (__main__.SelfTests.test_documentation_paths_do_not_hide_build_or_fixture_changes) ... ok
test_e2e_scope_keeps_production_and_shared_configuration_full (__main__.SelfTests.test_e2e_scope_keeps_production_and_shared_configuration_full) ... ok
test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies (__main__.SelfTests.test_e2e_shortcut_rejects_new_direct_renamed_and_target_dependencies) ... ok
test_e2e_shortcut_traverses_implicit_path_dependencies (__main__.SelfTests.test_e2e_shortcut_traverses_implicit_path_dependencies) ... ok
test_event_contract_requires_complete_candidate_and_optional_lanes (__main__.SelfTests.test_event_contract_requires_complete_candidate_and_optional_lanes) ... ok
test_every_wrong_result_missing_job_or_selection_fails_closed (__main__.SelfTests.test_every_wrong_result_missing_job_or_selection_fails_closed) ... ok
test_full_e2e_gate_preserves_workflow_branch_and_event_scope (__main__.SelfTests.test_full_e2e_gate_preserves_workflow_branch_and_event_scope) ... ok
test_git_range_includes_deleted_source_and_rename_origins (__main__.SelfTests.test_git_range_includes_deleted_source_and_rename_origins) ... ok
test_job_ids_and_display_names_cannot_hide_validation (__main__.SelfTests.test_job_ids_and_display_names_cannot_hide_validation) ... ok
test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate (__main__.SelfTests.test_report_only_stack_lane_mirrors_e2e_smoke_outside_the_aggregate) ... ok
test_repository_wiring_and_missing_dependency_regression (__main__.SelfTests.test_repository_wiring_and_missing_dependency_regression) ... ok
test_verify_command_preserves_failures (__main__.SelfTests.test_verify_command_preserves_failures) ... ok

----------------------------------------------------------------------
Ran 13 tests in 3.280s

OK
security coverage self-test passed
test_api_failures_preserve_other_evidence_and_never_enter_grace (__main__.SelfTests.test_api_failures_preserve_other_evidence_and_never_enter_grace) ... ok
test_attempt_outcome_does_not_replace_recent_success (__main__.SelfTests.test_attempt_outcome_does_not_replace_recent_success) ... ok
test_cancelled_attempt_cannot_refresh_expired_success (__main__.SelfTests.test_cancelled_attempt_cannot_refresh_expired_success) ... ok
test_cli_requires_the_repository_default_branch (__main__.SelfTests.test_cli_requires_the_repository_default_branch) ... ok
test_config_rejects_duplicate_and_invalid_entries (__main__.SelfTests.test_config_rejects_duplicate_and_invalid_entries) ... ok
test_freshness_boundaries (__main__.SelfTests.test_freshness_boundaries) ... ok
test_grace_requires_two_successful_queries_with_no_history (__main__.SelfTests.test_grace_requires_two_successful_queries_with_no_history) ... ok
test_invalid_api_evidence_fails_closed (__main__.SelfTests.test_invalid_api_evidence_fails_closed) ... ok
test_report_retains_every_workflow (__main__.SelfTests.test_report_retains_every_workflow) ... ok
test_requests_filter_schedule_default_branch_and_success_on_server (__main__.SelfTests.test_requests_filter_schedule_default_branch_and_success_on_server) ... ok

----------------------------------------------------------------------
Ran 10 tests in 0.051s

OK
..............................
----------------------------------------------------------------------
Ran 30 tests in 22.354s

OK
..........
----------------------------------------------------------------------
Ran 10 tests in 2.969s

OK
.................................
----------------------------------------------------------------------
Ran 33 tests in 0.160s

OK
Online functional runner labels: pf-testing
.......
----------------------------------------------------------------------
Ran 7 tests in 0.002s

OK
.......
----------------------------------------------------------------------
Ran 7 tests in 0.001s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.128s

OK
...
----------------------------------------------------------------------
Ran 3 tests in 0.547s

OK
PASS: cold stampede script self-test
exit_code=0

$ python3 ./scripts/test_object_generation_protocol_model.py
test_a_recovery_barrier_cannot_choose_an_empty_value (__main__.ObjectGenerationProtocolModel.test_a_recovery_barrier_cannot_choose_an_empty_value) ... ok
test_all_bounded_two_proposer_interleavings_choose_at_most_one_value (__main__.ObjectGenerationProtocolModel.test_all_bounded_two_proposer_interleavings_choose_at_most_one_value) ... ok
test_an_unavailable_decision_quorum_cannot_choose_a_candidate (__main__.ObjectGenerationProtocolModel.test_an_unavailable_decision_quorum_cannot_choose_a_candidate) ... ok
test_counter_allocation_requires_persistence_and_is_consumed_once (__main__.ObjectGenerationProtocolModel.test_counter_allocation_requires_persistence_and_is_consumed_once) ... ok
test_counter_restart_burns_persisted_allocations_before_or_after_issue (__main__.ObjectGenerationProtocolModel.test_counter_restart_burns_persisted_allocations_before_or_after_issue) ... ok
test_crash_after_promise_before_reply_does_not_restore_an_older_ballot (__main__.ObjectGenerationProtocolModel.test_crash_after_promise_before_reply_does_not_restore_an_older_ballot) ... ok
test_duplicate_or_unconfigured_voters_cannot_form_a_quorum (__main__.ObjectGenerationProtocolModel.test_duplicate_or_unconfigured_voters_cannot_form_a_quorum) ... ok
test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision (__main__.ObjectGenerationProtocolModel.test_empty_or_unobserved_minority_slot_can_retry_without_allocating_another_revision) ... ok
test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected (__main__.ObjectGenerationProtocolModel.test_equal_ballot_cannot_accept_different_bytes_and_lower_ballot_is_rejected) ... ok
test_forgetting_tombstone_authority_allows_an_aba_create (__main__.ObjectGenerationProtocolModel.test_forgetting_tombstone_authority_allows_an_aba_create) ... ok
test_issuing_before_persistence_reuses_a_ballot_after_restart (__main__.ObjectGenerationProtocolModel.test_issuing_before_persistence_reuses_a_ballot_after_restart) ... ok
test_lost_ack_and_restart_preserve_the_chosen_operation (__main__.ObjectGenerationProtocolModel.test_lost_ack_and_restart_preserve_the_chosen_operation) ... ok
test_minority_accepted_work_is_adopted_and_finished (__main__.ObjectGenerationProtocolModel.test_minority_accepted_work_is_adopted_and_finished) ... ok
test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample (__main__.ObjectGenerationProtocolModel.test_omitting_highest_accepted_adoption_has_a_double_choice_counterexample) ... ok
test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it (__main__.ObjectGenerationProtocolModel.test_quorum_choice_does_not_require_an_uncontacted_disk_to_learn_it) ... ok
test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor (__main__.ObjectGenerationProtocolModel.test_recovery_failure_or_stale_expected_head_does_not_allocate_a_successor) ... ok
test_replacing_voters_with_empty_state_can_choose_a_conflicting_value (__main__.ObjectGenerationProtocolModel.test_replacing_voters_with_empty_state_can_choose_a_conflicting_value) ... ok
test_skipping_predecessor_recovery_chooses_a_broken_revision_chain (__main__.ObjectGenerationProtocolModel.test_skipping_predecessor_recovery_chooses_a_broken_revision_chain) ... ok
test_stale_publication_rollback_and_gc_cannot_change_the_winner (__main__.ObjectGenerationProtocolModel.test_stale_publication_rollback_and_gc_cannot_change_the_winner) ... ok
test_superseded_operation_identity_cannot_be_reused_for_different_input (__main__.ObjectGenerationProtocolModel.test_superseded_operation_identity_cannot_be_reused_for_different_input) ... ok
test_superseded_operation_retry_returns_its_original_successor (__main__.ObjectGenerationProtocolModel.test_superseded_operation_retry_returns_its_original_successor) ... ok
test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision (__main__.ObjectGenerationProtocolModel.test_tombstone_and_null_marker_payloads_do_not_reset_authority_revision) ... ok
test_tombstone_rejects_a_delayed_never_present_create (__main__.ObjectGenerationProtocolModel.test_tombstone_rejects_a_delayed_never_present_create) ... ok
test_unlearned_predecessor_is_recovered_before_the_second_slot (__main__.ObjectGenerationProtocolModel.test_unlearned_predecessor_is_recovered_before_the_second_slot) ... ok

----------------------------------------------------------------------
Ran 24 tests in 0.958s

OK
model: states=51023, transitions=167520, terminal=732, chosen_candidates=2
negative control: double-choice trace=((0, 'prepare', 0), (0, 'prepare', 1), (0, 'prepare', 2), (0, 'accept', 0), (0, 'accept', 1), (0, 'accept', 2), (1, 'prepare', 0), (1, 'prepare', 1), (1, 'prepare', 2), (1, 'accept', 0), (1, 'accept', 1), (1, 'accept', 2))
multi-slot round schedules: 640
exit_code=0

$ python3 ./scripts/check_test_wiring.py
OK: e2e modules, runner selection, fuzz matrices, profiles, scheduled alerts, and Quick Checks are wired
exit_code=0

$ ./scripts/check_no_planning_docs.sh
exit_code=0

$ ./scripts/check_uring_lane_lib_only.sh
OK: no 'uring_'-matching test functions under crates/ecstore/tests; --lib is safe
exit_code=0

Initial complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID e07fc5b9-c6c8-4cdd-ae61-54839bf24437 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.016s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.011s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.009s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  16.950s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.025s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.008s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.007s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.035s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  21.087s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Restored complete gate
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
 Nextest run ID 233b7307-5361-4eb7-94d5-b384f17d2d6a with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.009s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.008s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.008s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.008s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  19.518s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   4.146s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.009s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.018s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.009s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.054s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  23.787s] 10 tests run: 10 passed, 0 skipped
exit_code=0

Mutation M1: success_measurement reports max(actual elapsed, requested budget + 1), forcing 1001 ms for the one-second request. The retained duration assertion at rustfs/tests/connect_perf_drive.rs:105 fails.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 06s
────────────
 Nextest run ID 9f618777-7bf6-4cbc-8803-dd7f50f449ed with nextest profile: ci
    Starting 1 test across 1 binary (9 tests skipped)
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (106986846) panicked at rustfs/tests/connect_perf_drive.rs:105:5:
    assertion failed: (1..=1_000).contains(&data.duration_millis)
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [   0.017s] 1 test run: 0 passed, 1 failed, 9 skipped
        FAIL [   0.016s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Mutation M2: target_parameters reports requested duration + 1. The in-process assertion at rustfs/tests/connect_perf_drive.rs:111 and CLI JSON assertion at rustfs/tests/connect_perf_drive.rs:527 both reject 1001 against 1000.
$ cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast -E 'test(real_local_write_read_is_measured_and_scratch_is_removed) | test(production_cli_measures_and_signs_exact_binary_provenance)'
info: experimental features enabled: setup-scripts
   Compiling rustfs v1.0.1 (/private/tmp/gw-orchestrator-2734/ci-drive/rustfs)
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs") generated 1 warning
warning: `rustfs` (bin "rustfs-cli") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 7m 20s
────────────
 Nextest run ID b73d46ab-c910-4e1e-8063-7fc7e332ec15 with nextest profile: ci
    Starting 2 tests across 1 binary (8 tests skipped)
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
  stdout ───

    running 1 test
    test production_cli_measures_and_signs_exact_binary_provenance ... FAILED

    failures:

    failures:
        production_cli_measures_and_signs_exact_binary_provenance

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 13.96s

  stderr ───

    thread 'production_cli_measures_and_signs_exact_binary_provenance' (107015462) panicked at rustfs/tests/connect_perf_drive.rs:527:5:
    assertion `left == right` failed
      left: Number(1001)
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
  stdout ───

    running 1 test
    test real_local_write_read_is_measured_and_scratch_is_removed ... FAILED

    failures:

    failures:
        real_local_write_read_is_measured_and_scratch_is_removed

    test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 9 filtered out; finished in 0.01s

  stderr ───

    thread 'real_local_write_read_is_measured_and_scratch_is_removed' (107016667) panicked at rustfs/tests/connect_perf_drive.rs:111:5:
    assertion `left == right` failed
      left: 1001
     right: 1000
    note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

────────────
     Summary [  13.985s] 2 tests run: 0 passed, 2 failed, 8 skipped
        FAIL [  13.968s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        FAIL [   0.015s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
error: test run failed
exit_code=100

Transient production mutations were restored byte-for-byte before the complete green rerun. Original/restored SHA256: 8d389e6d8836c041aab8c6a93e56addd65ba3f70b33264e1c74f639d0572a2da.

$ git diff --check

exit_code=0

$ git diff --cached --check

exit_code=0

Independent orchestrator acceptance
Tested code tree: e67c72fb01

Tested tree: 16fbbd2ea4
shellcheck --version && actionlint -> exit 0
cargo fmt --all --check -> exit 0
./scripts/check_unsafe_code_allowances.sh -> exit 0
./scripts/check_layer_dependencies.sh -> exit 0
./scripts/check_architecture_migration_rules.sh -> exit 0
./scripts/check_logging_guardrails.sh -> exit 0
./scripts/check_error_other_format_ratchet.sh -> exit 0
./scripts/check_no_tokio_io_uring.sh -> exit 0
./scripts/check_extension_schema_boundaries.sh -> exit 0
./scripts/check_body_cache_whitelist.sh -> exit 0
./scripts/check_s3s_footprint.sh -> exit 0
./scripts/check_test_count_ratchet.sh -> exit 0
./scripts/check_fips_wording.sh -> exit 0
./scripts/check_embedded_secrets.sh -> exit 0
make script-tests -> exit 0
python3 ./scripts/test_object_generation_protocol_model.py -> exit 0
python3 ./scripts/check_test_wiring.py -> exit 0
./scripts/check_no_planning_docs.sh -> exit 0
./scripts/check_uring_lane_lib_only.sh -> exit 0

$ LANG=en_US.UTF-8 CARGO_BUILD_JOBS=1 CARGO_INCREMENTAL=0 cargo nextest run --profile ci -p rustfs --test connect_perf_drive --no-fail-fast
info: experimental features enabled: setup-scripts
warning: linker stderr: ld: __eh_frame section too large (max 16MB) to encode dwarf unwind offsets in compact unwind table, performance of exception handling might be affected
  |
  = note: `#[warn(linker_messages)]` on by default

warning: `rustfs` (bin "rustfs-cli") generated 1 warning
warning: `rustfs` (bin "rustfs") generated 1 warning (1 duplicate)
    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.69s
────────────
 Nextest run ID cd1edf4c-f1c0-456d-ad1a-76cc00ce05c9 with nextest profile: ci
    Starting 10 tests across 1 binary
        PASS [   0.012s] rustfs::connect_perf_drive cancellation_and_single_collector_leave_no_scratch
        PASS [   0.010s] rustfs::connect_perf_drive cleanup_failure_overrides_cancellation_and_remains_attributed
        PASS [   0.010s] rustfs::connect_perf_drive consent_and_resource_budgets_fail_before_scratch_io
        PASS [   0.009s] rustfs::connect_perf_drive invalid_roots_and_existing_scratch_fail_without_clobbering
        PASS [  20.328s] rustfs::connect_perf_drive production_cli_measures_and_signs_exact_binary_provenance
        PASS [   5.527s] rustfs::connect_perf_drive production_cli_rejects_missing_l1_consent_and_limits_before_artifacts
        PASS [   0.013s] rustfs::connect_perf_drive read_only_root_and_full_drive_errors_are_classified
        PASS [   0.019s] rustfs::connect_perf_drive real_local_write_read_is_measured_and_scratch_is_removed
        PASS [   0.011s] rustfs::connect_perf_drive runtime_timeout_and_full_drive_use_schema_valid_failed_reason
        PASS [   0.049s] rustfs::connect_perf_drive successful_result_has_signed_bounded_private_offline_export
────────────
     Summary [  25.991s] 10 tests run: 10 passed, 0 skipped

exit_code=0

All 54 tracked Cargo.toml files parse without duplicate keys. git diff --check passed. The production source remains unchanged; the two deliberate implementation mutations above were restored before this complete green run.
2026-10-09 08:21:15 +08:00
Chris 4d15b81623 refactor(protocols): move error types to rustfs-s3-types by codemod (#8439)
T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.

- rustfs/src/protocols/client.rs: the two webdav session methods now
  return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
  gets its own `use rustfs_s3_types::s3_error;` once its parent imports
  that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
  edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).
2026-10-08 15:59:34 -07:00
Chris 836d30192f feat(s3-types): add Body::from_stream and move Swift off s3s::Body (#8438)
rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.

The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).
2026-10-08 15:05:16 -07:00
Chris 860c61ee6d feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor (#8437)
* feat(app): add the RustFS-owned RequestEnvelope and its s3s constructor

* ci: classify the s3s edge adapters in storage/s3_compat as interface
2026-10-08 11:58:55 -07:00
Chris f600af4704 test(e2e): compare two RUSTFS_S3_STACK servers over every S3 operation (#8435)
Add dual_stack_differential to e2e_test. It starts two rustfs processes
from the same binary, stack A on RUSTFS_S3_STACK=legacy and stack B on
RUSTFS_E2E_STACK_B (default legacy until rustfs/backlog#2734 T2.5),
sends both the same aws-sdk-s3 script covering the 73 methods of
impl S3 for FS (a normal request and a 4xx negative each), and compares
status, error code, an allowlist of response headers and a few typed
body fields. Error message text is never compared. Each run writes
<target>/dual-stack/<run>.json with compared_ops and the (op, field,
a, b) differences; any difference fails the test.

differential_detects_an_injected_difference starts B with
RUSTFS_REGION=eu-west-1 and requires GetBucketLocation to be reported.
The comparator is pure and unit-tested without a server.

Wire the module into the e2e-smoke filter and update the e2e-smoke and
e2e-full selection digests and the e2e_test test-count row.

Refs rustfs/backlog#2740
2026-10-08 10:47:10 -07:00
overtrue 9e747e8475 ci(footprint): count only s3_error! lines still on s3s's macro
s3_error_lines counted every line holding the text `s3_error!`, which
also matches the RustFS-owned `rustfs_s3_types::s3_error!`. A codemod
that moves call sites onto rustfs-s3-types therefore could not lower it
(rustfs/backlog#2743, decision 3; rustfs/rustfs#8432, gotcha 2).

The counter now applies a textual, per-file and per-line rule that errs
on the side of counting (full rule in the script header):
- occurrences spelled `rustfs_s3_types::s3_error!` never count;
- a line holding `s3s::s3_error!` always counts;
- any other `s3_error!` line counts unless its file is cleared, that is,
  it imports the macro from rustfs_s3_types by name on a line starting
  with the use item (or defines it with `macro_rules! s3_error`) and does
  not also bring s3s's macro in (`use s3s::s3_error`, a `use s3s::{..}`
  tree naming it or a crate-root `*`, `use s3s::*`, or
  `#[macro_use] extern crate s3s`);
- fail closed: a file matching neither form (`use super::*`, a
  crate-local re-export, a renamed or glob rustfs_s3_types import, a
  commented-out import, a comment-only mention) counts.

Census of the 1585 lines on 96fc06776: 1478 in 64 files that import the
macro from s3s; 107 in 15 files matching neither form (105 through
`use super::*`, 1 qualified `s3s::s3_error!`, 1 comment line); none in
files that import it from rustfs_s3_types or define it. Every line stays
counted, so the baseline remains s3_error_lines=1585 (files=202 and
ecstore_files=36 unchanged). Rewriting rustfs/src/auth.rs's import to
rustfs_s3_types lowers the new counter to 1567; the old one stayed 1585.

`--self-test` (wired into `make script-tests`) copies the script into
22 fixture trees: 6 positive cases and 16 negative ones (lines that must
not count, forms that must not clear a file, and a missing baseline,
missing key or failing classification search that must fail). Each tree
has a control line that must count, and each run gets s3s text on stdin.
Seven cases were red against the old counter. 19 mutants each turn their
case red: old definition, flipped import test, dropped fail-closed
branch, dropped e2e_test exclusion, dropped s3s veto, dropped qualified
s3s rule, dropped rustfs-qualified strip, unanchored rustfs import,
renamed import accepted, nested glob taken as root glob, single-line
s3s trees only, dropped s3s glob forms, dropped macro_use form, dropped
definition clearing, ignored classification rg errors, error-line search
without its path, classification searches without their path, missing
baseline skipped, missing key skipped.
2026-10-09 01:03:21 +08:00
Chris 96fc067765 refactor(ecstore): move S3ErrorCode to rustfs-s3-types by codemod (#8432)
* test(ecstore): pin the InvalidRange responses of new_getobjectreader

Both error branches of new_getobjectreader build an ErrorResponse with S3ErrorCode::InvalidRange, and no test observed either code. Pin the code, its wire name and the location fields of each branch before the code type moves to rustfs-s3-types (rustfs/backlog#2743), so a changed code on either branch fails a test. The ecstore test-count row rises by the two tests.

* chore(scripts): add the s3s-to-rustfs-s3-types error codemod

Task T1.2 of rustfs/backlog#2734 (rustfs/backlog#2743) moves S3Error, S3ErrorCode, S3Result and s3_error! from s3s to rustfs-s3-types by script, one crate closure per PR. The codemod rewrites use trees (splitting mixed s3s imports) and qualified code paths, never comments or literals, adds the rustfs-s3-types dependency, drops s3s only from a crate that no longer names it, refuses every file of the s3s edge allowlist, takes --hold for a file whose type is fixed outside the closure, and prints a per-crate census. A second run changes nothing. Its self-test runs under make script-tests.

* refactor(ecstore): take S3ErrorCode from rustfs-s3-types

Output of scripts/codemods/s3_error_to_s3_types.py over crates/ecstore, crates/lifecycle, crates/scanner and crates/replication (rustfs/backlog#2743 batches 1 and 2). The ErrorResponse code field, the two InvalidRange constructions in new_getobjectreader and an unused tier import now use rustfs_s3_types::S3ErrorCode; the variant names and wire names are unchanged. lifecycle, scanner and replication reference none of the four error names. warm_backend.rs and warm_backend_s3.rs are held: they compare against rustfs_s3_client::ErrorResponse.code, which stays the s3s type until s3-client moves.
2026-10-08 09:30:37 -07:00
Chris e809b4899e refactor(s3-types): own the HTTP Body and drop s3s from three crates (#8431)
Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.

The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.

rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.

Refs rustfs/backlog#2747, rustfs/backlog#2734
2026-10-08 08:43:06 -07:00
Chris 584f03f300 test(ecstore): reach the bucket-config codec through the test facade (#8428) 2026-10-08 07:12:19 -07:00
Chris d52d6fade7 ci: make the s3s footprint baseline exact and lower-only (#8403)
* ci: make the s3s footprint baseline exact and lower-only

The s3s footprint guard compared three counters against ceilings held in
the script itself, so a ceiling could sit above the tree and a drift was
fixable by raising it (rustfs/rustfs#5964 raised the s3_error! ceiling
to go green). The three values now live in
.config/s3s-footprint-baseline.txt as files, s3_error_lines and
ecstore_files, the script reads only that file, and a measurement that
differs from its baseline in either direction fails: growth prints
"s3s footprint grew: <measured> > <baseline>", shrinkage prints
"lower the baseline to <measured> in this PR". The counting method is
unchanged; the baseline is the current measurement of
gateway/integration (209 / 1585 / 36). The header records the end state:
once all three reach 0, T4.2 switches the script to failing on any
occurrence.

Refs rustfs/backlog#2739, rustfs/backlog#2734

* ci: add the s3s edge allowlist mode to the footprint guard (#8405)

`scripts/check_s3s_footprint.sh --mode allowlist` fails on every Rust
file that references s3s (`s3s::` or `s3s_sigv4`) and is not matched by
an entry of the new `.config/s3s-edge-allowlist.txt`; `--dry-run` lists
those files and exits 0. The allowlist carries the initial edge from the
spec, and the script rejects an entry that names or matches anything
under crates/ other than crates/e2e_test. The baseline mode and the
argument-less CI invocation are unchanged.

Refs rustfs/backlog#2741, rustfs/backlog#2734
2026-10-08 06:48:57 -07:00
Chris 3623d6db2d ci: raise the test-count baseline to the integration tree (#8426) 2026-10-08 06:48:40 -07:00
Chris 484f1254ca test(ecstore): pin the persisted bucket-config XML bytes for 13 families (#8408)
Before any DTO or codec is replaced, pin the exact bytes that
deserialize followed by serialize produces for every persisted
bucket-configuration XML family in its three source shapes (AWS
documentation, MinIO-written, RustFS-written), and the bytes RustFS
writes today for a typed value. Byte equality only: a structural
comparison would let a codec replacement change the persisted encoding
unnoticed.

The generator script refuses to overwrite existing goldens without
--force, refuses on dirty codec paths, cannot report success from its
write mode, and records the generating commit in the fixtures README.

Refs rustfs/backlog#2744, rustfs/backlog#2734
2026-10-08 21:25:56 +08:00
Chris c9c16efbb6 ci: add the per-crate test-count ratchet (#8404)
Quick Checks now fails any PR that lowers the number of test attributes
in a workspace crate or adds #[ignore], measured per crate against the
exact rows of .config/test-count-baseline.txt (rustfs/backlog#2734
section 1 rule 3). The counter widens the spec's literal
#[(tokio::)?test] to the tokio attributes that carry arguments and both
spellings of #[test_case(...)], and anchors at line start so
commented-out attributes do not count. The initial baseline covers 52
crates, 21077 tests and 164 ignored.

Refs rustfs/backlog#2738
Refs rustfs/backlog#2734
2026-10-08 21:25:42 +08:00
Chris f2a59073ca feat(s3-types): add the RustFS-owned S3 error type and s3_error! macro (#8406)
`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.

Refs rustfs/backlog#2742, rustfs/backlog#2734
2026-10-08 04:11:00 -07:00
Chris c1af94d1d1 refactor(server): group the s3s-compat tower layers under legacy_compat (#8410)
Move the ten tower layers that patch s3s behaviour out of
rustfs/src/server/layer.rs into rustfs/src/server/legacy_compat/, one file
per layer with its tests. Each file header names the s3s behaviour it
patches, the gateway mechanism that replaces it (Replaced by:), and the task
that removes it (Removed by: T2.12, rustfs/backlog#2771). The layers keep
their code, their order in external_service_stack!, and their behaviour; no
layer is deleted.

Add server::http::tests::stack_census, which renders the ordered service
chain of the real external_service_stack! expansion and compares it with a
committed constant, so a reorder, insertion or removal is a visible diff.

Route the two s3s error types the moved layers use through a
storage_api::server::legacy_compat shim so the split adds no s3s-importing
file, and lower the s3s footprint baselines to the measured values (files 209
to 208, s3_error! lines 1586 to 1585). Keep the server runtime-source bypass
guard covering the moved code.

Refs rustfs/backlog#2787, rustfs/backlog#2734
2026-10-08 02:57:47 -07:00
Chris f84bbfe28f feat(server): add the RUSTFS_S3_STACK selector and dual-stack e2e lane (#8411) 2026-10-07 21:01:39 -07:00
rustfs-cla-assistant[bot] 378a2ff7f8 Merge remote-tracking branch 'origin/main' into gateway/integration 2026-10-08 02:18:57 +00:00
rustfs-cla-assistant[bot] b88b53aa6a Merge remote-tracking branch 'origin/main' into gateway/integration 2026-10-07 21:18:17 +00:00
615 changed files with 38124 additions and 28357 deletions
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=7c705942d3a9e583ebaa85845daa21c4f81fdbf36def24db931aaef38636c53e
sha256-linux=5c33dc395233671cb77452f8fb12900ecdbeb368632222b96835d310bce1aab8
sha256-darwin=619402ae972a1b97d8684bb3c264091cb77f73d9fbdd59b36b1e002c28765f6b
sha256-linux=4359905a64855386a5403800c678cbd43dd2a5fda4777b8651aa213fcc66e585
+1 -1
View File
@@ -1 +1 @@
sha256=e4ee3898b840a8374cb02c64342cc15f8d7ecd1becaf4b4a2a4657290f9e7ff1
sha256=4eac269f9a59b2f9be4e7a87ea6aa38454f5ebe10ede35c9b8d4218137d2cd35
+1 -1
View File
@@ -1 +1 @@
sha256=00f8a56b75990b487a594b97c36610b94476a6944ba00473e37187ea654d42bf
sha256=879f421f1115d44f930e229510a2e533cfd5ebd4effe38730b82ef61f4fd7a86
+5
View File
@@ -70,6 +70,11 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
@echo "📦 Checking s3s footprint ratchet..."
./scripts/check_s3s_footprint.sh
.PHONY: test-count-ratchet-check
test-count-ratchet-check: ## Check no workspace crate lost tests or gained #[ignore] against the committed baseline
@echo "🧪 Checking per-crate test-count ratchet..."
./scripts/check_test_count_ratchet.sh
.PHONY: fips-wording-check
fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
@echo "📣 Checking cryptographic capability wording guard..."
+2 -2
View File
@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
./scripts/check_no_planning_docs.sh
.PHONY: pre-commit
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
@echo "✅ All pre-commit checks passed!"
.PHONY: pre-pr
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
@echo "✅ All pre-PR checks passed!"
.PHONY: dev-check
+2
View File
@@ -51,7 +51,9 @@ script-tests: ## Run shell script tests
./scripts/test_python_bin.sh
$(RUSTFS_PYTHON_BIN) ./scripts/test_check_doc_paths.py
./scripts/check_embedded_secrets.sh --self-test
./scripts/check_s3s_footprint.sh --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
+23 -8
View File
@@ -234,7 +234,7 @@ test-group = 'embedded-test-ports'
# processes cannot exhaust those budgets. Keep deadlines and assertions intact.
# The short HTTP fixtures are tracked in rustfs/backlog#2714.
[[profile.default.overrides]]
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|cleanup_accepts_missing_object_and_still_deletes_the_bucket|cleanup_diagnostics_identify_each_disconnected_transport_stage|cleanup_diagnostics_identify_each_failed_http_stage_without_private_context|cleanup_diagnostics_identify_timeouts_without_resetting_the_shared_deadline|oversized_response_is_rejected_without_buffering_it)$/)))'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
threads-required = "num-test-threads"
# This cancellation probe hashes the serving executable before acquiring its
@@ -356,6 +356,12 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-ecstore) & test(=store::init::tests::dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member)'
threads-required = "num-test-threads"
# These short-deadline real-disk progress probes need the test run's capacity
# for scanner and object-lock handoffs; keep their assertions and deadlines.
[[profile.default.overrides]]
filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
threads-required = "num-test-threads"
# ---------------------------------------------------------------------------
# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
# ---------------------------------------------------------------------------
@@ -424,6 +430,12 @@ threads-required = "num-test-threads"
filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
threads-required = "num-test-threads"
# Keep short-deadline real-disk progress probes from competing with unrelated
# tests for scanner and object-lock capacity. Assertions and deadlines stay on.
[[profile.ci.overrides]]
filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
threads-required = "num-test-threads"
# Emit this expensive fixture's stage timings in CI and use tmpfs when safe.
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(=store::init::tests::decommission_durable_ilm_receipt_pagination_fails_closed_on_second_page)'
@@ -548,7 +560,7 @@ filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnost
test-group = 'embedded-test-ports'
[[profile.ci.overrides]]
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|cleanup_accepts_missing_object_and_still_deletes_the_bucket|cleanup_diagnostics_identify_each_disconnected_transport_stage|cleanup_diagnostics_identify_each_failed_http_stage_without_private_context|cleanup_diagnostics_identify_timeouts_without_resetting_the_shared_deadline|oversized_response_is_rejected_without_buffering_it)$/)))'
filter = 'package(rustfs) & (binary(/^connect_registration$/) | binary(connect_inventory) | binary(connect_perf_drive) | (binary(connect_perf_client) & test(=configured_proxy_and_direct_endpoint_failures_are_distinct)) | (binary(connect_perf_object) & test(/^(real_rustfs_endpoint_and_production_cli_support_bounded_get_and_put|cleanup_uses_a_budget_independent_of_the_measurement_window|oversized_response_is_rejected_without_buffering_it)$/)))'
threads-required = "num-test-threads"
# Preserve the cancellation probe's startup budget and assertions in CI too.
@@ -698,6 +710,14 @@ test-group = 'ecstore-serial-flaky'
# the PR-lane e2e coverage for scanner usage snapshots consumed by quota and
# admin surfaces. It uses the same single-node, random-port, isolated-temp-dir
# fixture as the existing smoke modules.
#
# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
# starts two single-node servers from the same binary, one per
# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
# SDK script covering every operation of `impl S3 for FS` against both. Its
# comparator unit tests need no server. It meets the admission criteria
# unchanged; the module clause admits a new differential test without another
# filter edit.
[profile.e2e-smoke]
default-filter = """
package(e2e_test) & (
@@ -706,6 +726,7 @@ default-filter = """
| test(/^reliant::lifecycle::/)
| test(/^conditional_delete_objects_test::/)
| test(/^reliant::tiering::/)
| test(/^dual_stack_differential::/)
| test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
| test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
)
@@ -976,12 +997,6 @@ test-group = 'e2e-vault'
filter = 'package(e2e_test) & (test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/) | test(=reliant::tiering::test_hermetic_transition_restore_failure_expiry_and_retry))'
threads-required = "num-test-threads"
# The mc fixture uploads hundreds of objects concurrently before measuring listing.
# Reserve the runner so other server fixtures do not starve its disk operations.
[[profile.e2e-full.overrides]]
filter = 'package(e2e_test) & test(=mc_mirror_small_bucket_test::test_mc_mirror_small_bucket_completes_without_list_timeout)'
threads-required = "num-test-threads"
# Fixed, small contract for every executable server change. These existing
# scenarios exercise write/read, anonymous denial and retained bytes on restart.
[profile.core-contract]
-77
View File
@@ -1,77 +0,0 @@
{
"schema_version": 1,
"plan_id": "rustfs-release-validation",
"description": "Versioned certification plan for every public RustFS release exit (rustfs/backlog#2662, R01). preview, prerelease and stable all require this explicit plan. Scripts/release/release_gate.py fail-closes publication on any missing, duplicate, stale, substituted or non-PASS required item.",
"release_types": ["preview", "prerelease", "stable"],
"targets": [
{"os": "linux", "arch": "x86_64", "abi": "musl"},
{"os": "linux", "arch": "aarch64", "abi": "musl"},
{"os": "linux", "arch": "x86_64", "abi": "gnu"},
{"os": "linux", "arch": "aarch64", "abi": "gnu"},
{"os": "macos", "arch": "aarch64", "abi": "none"},
{"os": "windows", "arch": "x86_64", "abi": "msvc"}
],
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
"features": ["pyroscope"],
"trusted_workflows": [
{"repository": "rustfs/s3chaos", "workflow_id": "release-gate.yml"}
],
"trusted_workflows_basis": "The shared evidence envelope producer lives in rustfs/s3chaos (G01, fault/release_gate.rs) and its plan/live split workflow is .github/workflows/release-gate.yml there. Only results from these identities may enter a release certification; PR-uploaded JSON never gains release authority. R02 must extend this list when additional verifier workflows land, never widen it to untrusted sources.",
"lanes": [
{
"lane": "identity",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Fixed G01 evidence-envelope producer revision (rustfs/s3chaos PR #102 head recorded in rustfs/backlog#2658). R02 re-pins this when the live verifier ships.",
"timeout_seconds": 1800,
"environment_capabilities": ["process-exec"],
"cases": ["artifact-identity", "startup"],
"applies_to": {
"release_types": ["preview", "prerelease", "stable"],
"platforms": "all"
},
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
},
{
"lane": "upgrade",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Same fixed envelope producer as the identity lane; the workflow SHA and the product build SHA are bound separately and need not be equal.",
"timeout_seconds": 5400,
"environment_capabilities": ["process-exec", "previous-release-artifact"],
"cases": [
"direct-upgrade",
"mixed-version-upgrade",
"bucket-config-upgrade",
"bucket-config-rollback",
"multipart-layout-upgrade",
"multipart-layout-baseline"
],
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
"applies_to": {
"release_types": ["prerelease", "stable"],
"platforms": [{"os": "linux", "arch": "x86_64", "abi": "gnu"}]
},
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
}
],
"upgrade_sources": {
"basis": "Exactly the two previous-release sources pinned in .github/workflows/e2e-upgrade.yml are exercised today (source.version / source.asset / source.sha256). No other N-1/N-2 compatibility is claimed, and these digests are not inferred from tags.",
"sources": [
{
"version": "1.0.1",
"role": "N-1",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.1.zip",
"artifact_sha256": "9742e183cb0dfb0d48fa909797af572fe47b7f827569036baa4156e0d3958ee5"
},
{
"version": "1.0.0-rc.5",
"role": "N-2",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip",
"artifact_sha256": "3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d"
}
],
"rollback_supported": false,
"rollback_basis": "Only the bucket-config-rollback case (rollback_to_previous_release_reads_current_bucket_metadata) is exercised. Full data-plane rollback is not certified and is never a default claim."
}
}
+37
View File
@@ -0,0 +1,37 @@
# Files that may still reference s3s (`s3s::` or `s3s_sigv4`) when Phase 1 of
# the gateway migration ends (rustfs/backlog#2734 task T0.6, rustfs/backlog#2741).
# Consumed only by `scripts/check_s3s_footprint.sh --mode allowlist`.
#
# One entry per line: a repository-relative file path or a git `:(glob)`
# pathspec. `**` matches across directories, `*` stays within one path
# component, and a directory entry matches everything under it. Blank lines
# and lines starting with `#` are ignored; an entry may not contain whitespace,
# so a note belongs on its own comment line.
#
# Every Rust file listed by `git grep -l '\bs3s\(::\|_sigv4\)' -- '*.rs'` that
# no entry matches fails the allowlist mode. Task T1.9 switches CI to that
# mode; until then `--dry-run` prints the files still outside this list, which
# is the Phase 1 tracking list.
#
# Never add crates/** other than crates/e2e_test: the engine crates leave s3s
# entirely, and the script rejects such an entry. Only the s3s-facing edge of
# the rustfs binary and the e2e oracle keep it.
# rustfs binary: the s3s-facing edge that survives Phase 1
rustfs/src/storage/ecfs.rs
rustfs/src/storage/s3_compat/**
rustfs/src/storage/access.rs
rustfs/src/auth.rs
rustfs/src/error.rs
rustfs/src/storage_api.rs
rustfs/src/app/storage_api.rs
rustfs/src/admin/router.rs
rustfs/src/app/metadata_route.rs
rustfs/src/server/http.rs
rustfs/src/server/layer.rs
rustfs/src/server/hybrid.rs
rustfs/src/protocols/client.rs
# e2e oracle: the test target that verifies S3 behaviour against s3s
crates/e2e_test/src/fake_s3_target/**
crates/e2e_test/src/**
+17
View File
@@ -0,0 +1,17 @@
# Exact baseline for the direct s3s footprint, consumed only by
# scripts/check_s3s_footprint.sh (rustfs/backlog#1677; rustfs/backlog#2734 T0.4).
#
# Each value is the exact measurement of the tree, not a ceiling. CI fails when a
# measured counter differs from its line here in either direction:
# - a larger measurement means new code widened the s3s surface: route it
# through the gateway abstractions instead; never raise a value here;
# - a smaller measurement means the PR shrank the surface: lower the matching
# value to the measured number in the same PR.
#
# files Rust files referencing s3s paths, repo-wide minus crates/e2e_test
# s3_error_lines lines holding s3_error!, same scope, minus those the script
# header's rule clears (rustfs_s3_types's macro); fails closed
# ecstore_files Rust files referencing s3s paths under crates/ecstore/src
files=65
s3_error_lines=289
ecstore_files=0
+64
View File
@@ -0,0 +1,64 @@
# Per-crate test-count baseline consumed by scripts/check_test_count_ratchet.sh
# (rustfs/backlog#2734 section 1 rule 3; T0.3 in rustfs/backlog#2738).
#
# One row per workspace member: <package>, tests, ignored, tab-separated.
# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
# attribute lines under the package directory; ignored counts #[ignore] and
# #[ignore = "..."]. Both columns are exact: CI fails when a crate's tests fall
# below or rise above its row, or when its ignored count changes. Raise a row in
# the PR that adds tests; lowering one is a reviewed statement that tests were
# intentionally removed. Regenerate with:
# scripts/check_test_count_ratchet.sh --update
e2e_test 1012 20
rustfs 6157 64
rustfs-audit 70 0
rustfs-checksums 26 0
rustfs-common 36 0
rustfs-concurrency 87 0
rustfs-config 43 0
rustfs-connect-symbol-catalog 6 0
rustfs-credentials 24 0
rustfs-crypto 67 0
rustfs-data-usage 57 0
rustfs-ecstore 7561 2
rustfs-extension-schema 23 0
rustfs-filemeta 353 0
rustfs-heal 1097 3
rustfs-heal-contracts 8 0
rustfs-iam 401 0
rustfs-io-core 49 0
rustfs-io-metrics 154 0
rustfs-keystone 29 0
rustfs-kms 715 25
rustfs-license 2 0
rustfs-lifecycle 155 0
rustfs-lock 148 0
rustfs-log-analyzer 90 0
rustfs-madmin 210 0
rustfs-notify 140 0
rustfs-object-capacity 92 0
rustfs-object-data-cache 140 0
rustfs-obs 385 0
rustfs-policy 734 0
rustfs-protocols 748 1
rustfs-protos 73 0
rustfs-replication 209 0
rustfs-rio 218 0
rustfs-rio-v2 34 6
rustfs-s3-client 69 0
rustfs-s3-ops 8 0
rustfs-s3-types 105 0
rustfs-s3select-api 207 0
rustfs-s3select-query 204 2
rustfs-scanner 1340 20
rustfs-scanner-metrics 90 0
rustfs-security-governance 20 0
rustfs-signer 31 0
rustfs-storage-api 34 0
rustfs-targets 604 26
rustfs-test-utils 1 0
rustfs-tls-runtime 43 0
rustfs-trusted-proxies 68 0
rustfs-utils 247 0
rustfs-zip 30 0
zstd 11 0
+4
View File
@@ -96,6 +96,10 @@ runs:
shell: bash
run: ./scripts/check_s3s_footprint.sh
- name: Check per-crate test-count ratchet
shell: bash
run: ./scripts/check_test_count_ratchet.sh
- name: Check cryptographic capability wording
shell: bash
run: ./scripts/check_fips_wording.sh
-26
View File
@@ -296,32 +296,6 @@
"origin": "baseline",
"alert_contract": "scheduled-validation"
},
{
"workflow": ".github/workflows/rustfs-validation-sweep.yml",
"name": "RustFS Validation Sweep",
"roles": [
"periodic",
"manual"
],
"events": [
"schedule",
"workflow_dispatch"
],
"cadence": "nightly",
"timezone": "UTC",
"schedule": [
"30 19 * * *"
],
"owner": "@overtrue",
"policy": "active",
"monitor": "workflow",
"max_age_hours": 60,
"result_owner": "rustfs-validation-sweep.yml",
"disposition": "canonical validation cases executed in rotating batches",
"failure_owner": "workflow",
"origin": "baseline",
"alert_contract": "scheduled-validation"
},
{
"workflow": ".github/workflows/btf-prototype-reference.yml",
"name": "BTF prototype reference",
+5 -311
View File
@@ -814,18 +814,13 @@ jobs:
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Determine upload path based on build type. Release and prerelease
# builds only stage their bytes here: the official
# artifacts/rustfs/release/ path is written solely by the
# certification-gated promote-release-artifacts job, from these same
# staged objects. Development builds keep the uncertified dev channel
# and never touch official or latest channels.
# Determine upload path based on build type
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/dev/"
echo "📤 Uploading development build to R2 dev directory"
else
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
echo "📤 Staging release build for certification-gated promotion"
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"
echo "📤 Uploading release build to R2 release directory"
fi
# Upload all rustfs zip files to R2 using S3-compatible API
@@ -874,221 +869,6 @@ jobs:
echo "This run does not trigger Docker publishing."
fi
# Aggregate the parent candidate batch. This job always collects results and
# retains its evidence, including on failure, and every publication job
# below requires its success. Missing, stale, substituted or non-PASS
# evidence fails closed here: only the trusted verification sources
# (rustfs/s3chaos release-gate envelopes, rustfs/backlog#2658) can satisfy
# it, so a PR-uploaded JSON set never gains release authority.
release-certification:
name: Release Certification
needs: [ build-check, build-rustfs ]
if: ${{ !cancelled() && needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') }}
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
merge-multiple: true
- name: Write candidate and collect trusted validation results
env:
# Cross-repo read of the trusted verification run. Falls back to the
# workflow token, which cannot read rustfs/s3chaos; a missing token
# therefore fail-closes instead of certifying.
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
PRODUCT_COMMIT: ${{ github.sha }}
VERSION: ${{ needs.build-check.outputs.version }}
BUILD_TYPE: ${{ needs.build-check.outputs.build_type }}
FEATURES: ${{ needs.build-check.outputs.build_type == 'development' && '' || 'pyroscope' }}
shell: bash
run: |
set -euo pipefail
mkdir -p results release-gate-output
PLAN_SHA256=$(python3 -c "import hashlib,json,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('.config/release-validation-plan.json'))))")
python3 - "$PLAN_SHA256" > candidate.json <<'PY'
import hashlib, json, os, sys, zipfile
plan_sha256 = sys.argv[1]
profile = {"release": "stable", "prerelease": "prerelease", "preview": "preview"}.get(os.environ["BUILD_TYPE"])
if profile is None:
raise SystemExit("release certification only covers preview, prerelease and stable builds")
def sha256(path):
digest = hashlib.sha256()
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
deliverables = []
for name in sorted(os.listdir("artifacts")):
if not name.startswith("rustfs-") or not name.endswith(".zip"):
continue
path = os.path.join("artifacts", name)
members = zipfile.ZipFile(path).namelist()
binary_name = "rustfs.exe" if "rustfs.exe" in members else "rustfs"
with zipfile.ZipFile(path) as archive:
binary_bytes = archive.read(binary_name)
import tempfile
with tempfile.NamedTemporaryFile() as handle:
handle.write(binary_bytes)
handle.flush()
binary_sha = sha256(handle.name)
deliverables.append({
"kind": "zip",
"name": name,
"location": "https://github.com/rustfs/rustfs/releases/download/{}/{}".format(os.environ["VERSION"], name),
"sha256": sha256(path),
"size": os.path.getsize(path),
"binary_sha256": binary_sha,
})
if not deliverables:
raise SystemExit("no built deliverables found to certify")
candidate = {
"schema_version": 1,
"product_commit": os.environ["PRODUCT_COMMIT"],
"version": os.environ["VERSION"],
"release_profile": profile,
"features": [f for f in os.environ.get("FEATURES", "").split(",") if f],
"plan_sha256": plan_sha256,
"validation_batch": {"run_id": os.environ["GITHUB_RUN_ID"], "run_attempt": int(os.environ["GITHUB_RUN_ATTEMPT"])},
"deliverables": deliverables,
}
json.dump(candidate, sys.stdout, indent=2, sort_keys=True)
print()
PY
# Trusted verification evidence only: the release-gate envelopes of
# rustfs/s3chaos for this exact product commit. Absence blocks below.
TRUSTED_REPO="rustfs/s3chaos"
TRUSTED_WORKFLOW="release-gate.yml"
TRUSTED_RUN=$(gh api "repos/${TRUSTED_REPO}/actions/workflows/${TRUSTED_WORKFLOW}/runs?head_sha=${PRODUCT_COMMIT}&status=success&per_page=1" --jq '.workflow_runs[0].id // empty' || true)
if [[ -z "$TRUSTED_RUN" ]]; then
echo "❌ No trusted verification run of ${TRUSTED_REPO}/${TRUSTED_WORKFLOW} for ${PRODUCT_COMMIT}; refusing to certify" >&2
else
gh run download "$TRUSTED_RUN" -R "$TRUSTED_REPO" -n release-validation-evidence -D results || \
echo "❌ Trusted verification run ${TRUSTED_RUN} published no release-validation-evidence artifact" >&2
fi
python3 scripts/release/release_gate.py validate \
--plan .config/release-validation-plan.json \
--candidate candidate.json \
--results results \
--output release-gate-output
- name: Upload certification evidence
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-certification
path: |
candidate.json
results/
release-gate-output/
if-no-files-found: warn
retention-days: 30
# The only writer of the official R2 release path. Promotion re-checks the
# exact bytes about to be published against the passed certification list
# and copies the staged objects byte-for-byte: no rebuild-after-verify.
promote-release-artifacts:
name: Promote Release Artifacts
needs: [ build-check, build-rustfs, release-certification ]
# Target-channel lock for the official R2 release path.
concurrency:
group: release-channel-r2-release
cancel-in-progress: false
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download build and certification artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
merge-multiple: true
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify and promote staged release artifacts
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "❌ R2 credentials missing; refusing to promote to the official channel" >&2
exit 1
fi
for zip_file in artifacts/rustfs-*.zip; do
[[ -f "$zip_file" ]] || continue
name="$(basename "$zip_file")"
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$zip_file" \
--artifact-name "$name"
done
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
STAGING_PATH="s3://${R2_BUCKET}/artifacts/rustfs/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
RELEASE_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"
for zip_file in artifacts/rustfs-*.zip; do
[[ -f "$zip_file" ]] || continue
name="$(basename "$zip_file")"
staged="$(mktemp)"
aws s3 cp "${STAGING_PATH}${name}" "$staged" --endpoint-url "$R2_ENDPOINT" --only-show-errors
staged_sha="$(sha256sum "$staged" | awk '{print $1}')"
built_sha="$(sha256sum "$zip_file" | awk '{print $1}')"
if [[ "$staged_sha" != "$built_sha" ]]; then
echo "❌ Staged object ${name} does not match the verified bytes" >&2
exit 1
fi
aws s3 cp "$staged" "${RELEASE_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
rm -f "$staged"
echo "✅ Promoted ${name} (${built_sha})"
done
# Create GitHub Release for every valid release tag, including previews
create-release:
name: Create GitHub Release
@@ -1248,10 +1028,6 @@ jobs:
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
# Target-channel lock for the latest.json version pointer.
concurrency:
group: release-channel-oss-latest
cancel-in-progress: false
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
@@ -1292,45 +1068,6 @@ jobs:
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
chmod +x "$OSSUTIL_BIN"
# Candidate/pointer ordering: an older candidate must never overwrite
# a newer pointer, even if its promotion job finishes later.
CURRENT_LATEST="${RUNNER_TEMP}/current-latest.json"
if "$OSSUTIL_BIN" cp oss://rustfs-version/latest.json "$CURRENT_LATEST" 2>/dev/null; then
python3 - "$CURRENT_LATEST" "$VERSION" <<'PY'
import json
import re
import sys
def key(version):
match = re.match(r"^(\d+)\.(\d+)\.(\d+)(?:-([0-9A-Za-z.-]+))?$", version)
if not match:
raise SystemExit(3)
core = tuple(int(part) for part in match.groups()[:3])
prerelease = match.group(4)
if prerelease is None:
return core + (1,)
return core + (0, tuple(int(p) if p.isdigit() else p for p in re.split(r"[.-]", prerelease)))
try:
current = json.load(open(sys.argv[1], encoding="utf-8")).get("version", "")
proposed = sys.argv[2]
if current and key(current) > key(proposed):
raise SystemExit(1)
except SystemExit:
raise
except (OSError, ValueError, TypeError):
raise SystemExit(3)
PY
guard_status=$?
if [[ "$guard_status" -eq 1 ]]; then
echo "Current latest.json points at a newer version; refusing to downgrade" >&2
exit 0
elif [[ "$guard_status" -ne 0 ]]; then
echo "Existing latest.json is unparsable; refusing to overwrite it blindly" >&2
exit 1
fi
fi
# Create latest.json
cat > latest.json << EOF
{
@@ -1347,59 +1084,16 @@ jobs:
echo "✅ Updated latest.json for ${RELEASE_TYPE} release $VERSION"
# Publish release (remove draft status). Requires the release-certification
# aggregate: without a complete legal success there is no public write.
# Publish release (remove draft status)
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets, release-certification ]
# Target-channel lock: concurrent releases serialize on this channel so an
# older candidate cannot interleave its publish with a newer one.
concurrency:
group: release-channel-github-release
cancel-in-progress: false
needs: [ build-check, create-release, upload-release-assets ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify release assets against certification
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
set -euo pipefail
TAG="${{ needs.build-check.outputs.version }}"
mkdir -p "$RUNNER_TEMP/promotion-assets"
gh release download "$TAG" --dir "$RUNNER_TEMP/promotion-assets" --pattern '*.zip'
found=0
for zip_file in "$RUNNER_TEMP"/promotion-assets/*.zip; do
[[ -f "$zip_file" ]] || continue
found=1
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$zip_file" \
--artifact-name "$(basename "$zip_file")"
done
if [[ "$found" -ne 1 ]]; then
echo "No release zip assets to verify against the certification" >&2
exit 1
fi
- name: Publish release
env:
GH_TOKEN: ${{ github.token }}
+2 -2
View File
@@ -96,7 +96,7 @@ env:
RUST_BACKTRACE: 1
jobs:
# Readers: workspace-lint-and-build, workspace-tests (workspace and ILM), build-rustfs-debug-binary,
# Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
# e2e-tests, e2e-full.
warm-ci-dev:
name: Warm ci-dev
@@ -146,7 +146,7 @@ jobs:
if-no-files-found: error
# --all-targets covers the test binaries nextest builds, including
# e2e_test, which the workspace run excludes. The second build adds
# e2e_test, which test-and-lint's own run excludes. The second build adds
# the e2e-test-hooks feature resolution that build-rustfs-debug-binary uses
# and that no lint lane enables.
- name: Build ci-dev superset
+147 -41
View File
@@ -16,7 +16,9 @@ name: Continuous Integration
on:
push:
branches: [ main ]
# gateway/integration receives the daily merge of main as a direct push
# (rustfs/backlog#2734), so its pushes must prove the result like main's.
branches: [ main, gateway/integration ]
paths-ignore:
- "**.md"
- "docs/**"
@@ -113,7 +115,7 @@ jobs:
printf '%s\n' 'mode=full' >> "$GITHUB_OUTPUT"
echo "Base CI policy unavailable; running the full matrix."
fi
if ! grep -q '^plan=' "$GITHUB_OUTPUT" || ! grep -q '^plan_contract=2$' "$GITHUB_OUTPUT"; then
if ! grep -q '^plan=' "$GITHUB_OUTPUT" || ! grep -q '^rehearse_shards=' "$GITHUB_OUTPUT"; then
printf '%s\n' 'mode=full' 'connect=true' 'full_validation=true' 'rehearse_shards=false' 'packages=' 'plan=' >> "$GITHUB_OUTPUT"
echo "Base policy does not cover this workflow contract; retaining the full matrix."
fi
@@ -160,7 +162,7 @@ jobs:
- name: Run shared quick checks
uses: ./.github/actions/quick-checks
workspace-lint-and-build:
test-and-lint:
name: Workspace Lint and Build
if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
@@ -193,13 +195,13 @@ jobs:
- name: Prepare test evidence
run: |
mkdir -p artifacts/workspace
mkdir -p artifacts/test-and-lint
{
echo "run_id=${GITHUB_RUN_ID}"
echo "job=${GITHUB_JOB}"
echo "runner=${RUNNER_NAME}"
echo "started_at=$(date --utc --iso-8601=seconds)"
} > artifacts/workspace/run-metadata.txt
} > artifacts/test-and-lint/run-metadata.txt
# Clippy gates every event, pull requests included. When it ran only after
# merge, a lint regression merged freely and then failed every main run
@@ -233,11 +235,11 @@ jobs:
- name: Run documentation tests
run: |
mkdir -p artifacts/workspace
mkdir -p artifacts/test-and-lint
set +e
timeout --verbose --signal=TERM --kill-after=30s 15m \
python3 scripts/ci_test_artifact.py cargo doc --configuration workspace \
2>&1 | tee artifacts/workspace/doctest.log
2>&1 | tee artifacts/test-and-lint/doctest.log
status=${PIPESTATUS[0]}
{
echo "command=cargo test --all --exclude e2e_test --doc"
@@ -246,7 +248,7 @@ jobs:
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|rustdoc|target/.*/deps/' || true
} > artifacts/workspace/doctest-diagnostics.txt
} > artifacts/test-and-lint/doctest-diagnostics.txt
exit "${status}"
- name: Upload test reports and diagnostics
@@ -255,7 +257,7 @@ jobs:
with:
name: junit-test-and-lint-${{ github.run_number }}-${{ github.run_attempt }}
path: |
artifacts/workspace
artifacts/test-and-lint
retention-days: 3
if-no-files-found: error
@@ -268,7 +270,7 @@ jobs:
workspace-tests:
name: Workspace tests
needs: [ workspace-lint-and-build, classify-changes ]
needs: [ test-and-lint, classify-changes ]
# The archive retains compile-time fixture paths and the large test set
# requires the same runner image until the hosted-runtime pilot passes.
runs-on: sm-standard-4
@@ -293,14 +295,14 @@ jobs:
- name: Extract and list workspace tests
run: |
mkdir -p artifacts/workspace
mkdir -p artifacts/test-and-lint
cargo nextest list --profile ci --archive-file "$RUNNER_TEMP/workspace-tests.tar.zst" --workspace-remap "$GITHUB_WORKSPACE" --extract-to "$GITHUB_WORKSPACE" --extract-overwrite --message-format json \
> artifacts/workspace/core-test-listing.json
python3 scripts/check_test_wiring.py --check-core artifacts/workspace/core-test-listing.json
> artifacts/test-and-lint/core-test-listing.json
python3 scripts/check_test_wiring.py --check-core artifacts/test-and-lint/core-test-listing.json
- name: Run nextest tests
run: |
mkdir -p artifacts/workspace
mkdir -p artifacts/test-and-lint
rm -f target/nextest/ci/junit.xml
./scripts/ci/resource_sampler.sh start nextest
trap './scripts/ci/resource_sampler.sh stop' EXIT
@@ -309,7 +311,7 @@ jobs:
cargo nextest run --profile ci --cargo-metadata target/nextest/cargo-metadata.json --binaries-metadata target/nextest/binaries-metadata.json \
--workspace-remap "$GITHUB_WORKSPACE" --target-dir-remap "$GITHUB_WORKSPACE/target" \
--status-level all --final-status-level all \
2>&1 | tee artifacts/workspace/nextest.log
2>&1 | tee artifacts/test-and-lint/nextest.log
status=${PIPESTATUS[0]}
{
echo "command=workspace archive execution"
@@ -321,13 +323,13 @@ jobs:
echo
echo "Kernel OOM / kill events:"
dmesg -T 2>/dev/null | grep -iE 'oom|out of memory|killed process' | tail -20 || true
} > artifacts/workspace/nextest-diagnostics.txt
} > artifacts/test-and-lint/nextest-diagnostics.txt
exit "${status}"
- name: Verify workspace execution completeness
run: |
python3 scripts/ci_test_artifact.py evidence artifacts/workspace/core-test-listing.json target/nextest/ci/junit.xml
./scripts/check_migration_gate_count.sh evidence artifacts/workspace/core-test-listing.json target/nextest/ci/junit.xml
python3 scripts/ci_test_artifact.py evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
- name: Run ignored ILM integration tests serially
if: needs.classify-changes.outputs.mode == 'full'
@@ -358,7 +360,7 @@ jobs:
name: workspace-test-evidence-${{ github.run_attempt }}
path: |
target/nextest/ci/junit.xml
artifacts/workspace
artifacts/test-and-lint
artifacts/ilm-integration
retention-days: 3
if-no-files-found: error
@@ -368,7 +370,7 @@ jobs:
if: needs.classify-changes.outputs.rehearse_shards == 'true'
run: |
python3 scripts/ci_test_shards.py prepare \
--listing artifacts/workspace/core-test-listing.json \
--listing artifacts/test-and-lint/core-test-listing.json \
--junit target/nextest/ci/junit.xml --output artifacts/workspace-shard-plan
- name: Upload same-run shard plan
if: needs.classify-changes.outputs.rehearse_shards == 'true'
@@ -449,7 +451,7 @@ jobs:
connect-feature-boundaries:
name: Connect Feature Boundaries
if: needs.classify-changes.outputs.mode == 'full' && needs.classify-changes.outputs.connect != 'false' && contains(fromJSON('["schedule", "workflow_dispatch"]'), github.event_name)
if: needs.classify-changes.outputs.mode == 'full' && needs.classify-changes.outputs.connect != 'false' && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
@@ -499,13 +501,11 @@ jobs:
test-and-lint-rio-v2:
name: Test and Lint (rio-v2)
# Specialized boundary, hardware and alternate-feature lanes run only on
# the nightly schedule and manual dispatch. They do not gate a merge: the
# nightly run reports a regression within a day, and repeating these builds
# on every PR revision and merge group bought no earlier signal worth the
# runner occupancy. Workspace lint and tests, the core contract, the debug
# build, E2E smoke and the S3 lanes remain the pull-request gate.
if: needs.classify-changes.outputs.mode == 'full' && contains(fromJSON('["schedule", "workflow_dispatch"]'), github.event_name)
# Alternate feature configurations run on the nightly schedule, merge
# groups and manual dispatch. The default workspace remains the PR gate and
# the per-push main baseline: repeating four feature builds on every merge
# bought nothing the nightly run does not report within a day.
if: needs.classify-changes.outputs.mode == 'full' && contains(fromJSON('["merge_group", "schedule", "workflow_dispatch"]'), github.event_name)
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
@@ -542,12 +542,12 @@ jobs:
# --profile ci so the quarantine list (and its junit flaky markers)
# covers this leg too; the default profile is the local no-retry
# profile and silently ignored quarantined flakes here (rustfs#6703).
mkdir -p artifacts/workspace
mkdir -p artifacts/test-and-lint
rm -f target/nextest/ci/junit.xml
./scripts/ci/resource_sampler.sh start nextest-rio-v2
trap './scripts/ci/resource_sampler.sh stop' EXIT
cargo nextest run --profile ci -p rustfs -p rustfs-ecstore --features rio-v2 \
2>&1 | tee artifacts/workspace/nextest.log
2>&1 | tee artifacts/test-and-lint/nextest.log
cargo test -p rustfs --doc --features rio-v2
- name: Upload rio-v2 test reports and diagnostics
@@ -557,14 +557,14 @@ jobs:
name: junit-test-and-lint-rio-v2-${{ github.run_number }}-${{ github.run_attempt }}
path: |
target/nextest/ci/junit.xml
artifacts/workspace
artifacts/test-and-lint
retention-days: 3
if-no-files-found: error
test-and-lint-protocols:
name: "Test and Lint (${{ matrix.features.name }})"
# Same cadence as test-and-lint-rio-v2 above.
if: needs.classify-changes.outputs.mode == 'full' && contains(fromJSON('["schedule", "workflow_dispatch"]'), github.event_name)
if: needs.classify-changes.outputs.mode == 'full' && contains(fromJSON('["merge_group", "schedule", "workflow_dispatch"]'), github.event_name)
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
@@ -633,7 +633,7 @@ jobs:
path: |
target/nextest/ci/junit.xml
artifacts/protocol-tests
artifacts/workspace
artifacts/test-and-lint
retention-days: 3
if-no-files-found: warn
@@ -785,7 +785,7 @@ jobs:
# job had neither, so each closed/merged PR really ran the whole io_uring
# suite (measured 4m17s / 7m19s / 7m31s on runs 30678272341 / 30678117601 /
# 30662728539) and kept the cancellation run in progress for minutes.
if: needs.classify-changes.outputs.mode == 'full' && contains(fromJSON('["schedule", "workflow_dispatch"]'), github.event_name)
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
# GitHub-hosted ubuntu-latest runs a recent kernel with io_uring and, unlike
# a container, applies no seccomp filter that would block io_uring_setup — so
@@ -899,6 +899,8 @@ jobs:
# Retain the producer image and fixture paths until the runtime pilot passes.
runs-on: sm-standard-4
timeout-minutes: 30
env:
RUSTFS_S3_STACK: legacy
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -980,7 +982,7 @@ jobs:
path: |
${{ runner.temp }}/rustfs-e2e-smoke-logs/
${{ runner.temp }}/rustfs-e2e-smoke-list.json
artifacts/workspace
artifacts/test-and-lint
if-no-files-found: warn
- name: Upload e2e smoke JUnit report
@@ -1019,18 +1021,122 @@ jobs:
path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
retention-days: 3
# The same e2e-smoke suite against the same debug binary with the gateway
# S3 stack selected (rustfs/backlog#2734). Report-only: it is outside the
# Test and Lint aggregate and becomes required at T2.10. The test-binary
# download, identity, membership, run and completeness steps must stay
# identical to e2e-tests; scripts/ci_gate.py enforces that.
e2e-tests-gateway:
name: e2e-smoke (gateway, report-only until T2.10)
needs: [ build-rustfs-debug-binary ]
runs-on: sm-standard-2
timeout-minutes: 30
continue-on-error: true
env:
RUSTFS_S3_STACK: gateway
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install verification runtime
uses: ./.github/actions/test-runtime
- name: Download E2E test binaries
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: e2e-test-binaries
path: ${{ runner.temp }}
- name: Verify E2E archive identity
run: python3 scripts/ci_test_artifact.py verify --configuration e2e --archive "$RUNNER_TEMP/e2e-tests.tar.zst"
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install awscurl
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
- name: Verify awscurl
run: test -x "$AWSCURL_PATH"
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
- name: Download debug binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
# Build the e2e test graph once. The archive is reused by the smoke
# selection guard, security exact-count check, and run below, avoiding a
# second compile of the same e2e_test target on cold runners (backlog#1645).
- name: Verify e2e smoke test membership
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/e2e-tests.tar.zst
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
run: |
cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --workspace-remap "$GITHUB_WORKSPACE" --extract-to "$GITHUB_WORKSPACE" --extract-overwrite --message-format json > "${NEXTEST_LISTING}"
python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
# PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
# profile.e2e-smoke default-filter in .config/nextest.toml is the single
# wiring mechanism for e2e tests in CI — extend that filter instead of
# adding new e2e jobs here. Each test spawns its own rustfs server on a
# random port and reuses the downloaded debug binary above.
- name: Run e2e smoke suite
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/e2e-tests.tar.zst
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
SAMPLER_INTERVAL_SECS: "15"
run: |
./scripts/ci/resource_sampler.sh start e2e-smoke
trap './scripts/ci/resource_sampler.sh stop' EXIT
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --cargo-metadata target/nextest/cargo-metadata.json --binaries-metadata target/nextest/binaries-metadata.json --workspace-remap "$GITHUB_WORKSPACE" --target-dir-remap "$GITHUB_WORKSPACE/target" \
--status-level all --final-status-level all --failure-output final
- name: Verify e2e smoke execution completeness
run: python3 scripts/ci_test_artifact.py evidence "$RUNNER_TEMP/rustfs-e2e-smoke-list.json" target/nextest/e2e-smoke/junit.xml
- name: Upload e2e smoke diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-gateway-diagnostics-${{ github.run_number }}-${{ github.run_attempt }}
path: |
${{ runner.temp }}/rustfs-e2e-smoke-logs/
${{ runner.temp }}/rustfs-e2e-smoke-list.json
if-no-files-found: warn
- name: Upload e2e smoke JUnit report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-gateway-junit-${{ github.run_number }}-${{ github.run_attempt }}
path: target/nextest/e2e-smoke/junit.xml
if-no-files-found: warn
e2e-full:
name: End-to-End Tests (full merge gate)
# Merge gate only (backlog#1149 ci-5): the never-automated user-visible
# suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
# security-boundary, ... — via the e2e-full nextest profile. Too heavy for
# every PR, so it is gated to main pushes, the merge queue, and manual
# dispatch. protocols / the 7 cluster suites / replication / #[ignore] are
# owned by other lanes (see .config/nextest.toml profile.e2e-full).
# every PR, so it is gated to main and gateway/integration pushes, the merge
# queue, and manual dispatch. protocols / the 7 cluster suites / replication
# / #[ignore] are owned by other lanes (see .config/nextest.toml profile.e2e-full).
if: >-
github.event_name == 'workflow_dispatch' ||
github.event_name == 'merge_group' ||
(github.event_name == 'push' && github.ref == 'refs/heads/main')
(github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/gateway/integration'))
needs: [ build-rustfs-debug-binary ]
# Retain the producer image and fixture paths until the runtime pilot passes.
runs-on: sm-standard-4
@@ -1391,7 +1497,7 @@ jobs:
- classify-changes
- typos
- quick-checks
- workspace-lint-and-build
- test-and-lint
- workspace-tests
- workspace-shards
- workspace-shard-evidence
@@ -1427,7 +1533,7 @@ jobs:
- required-checks
- typos
- quick-checks
- workspace-lint-and-build
- test-and-lint
- workspace-tests
- core-contract
- test-and-lint-rio-v2
+2 -182
View File
@@ -329,75 +329,13 @@ jobs:
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Consume the trusted release certification for the exact version this
# workflow wraps and verify the certified input digests. A manual dispatch
# without that certification fails here and pushes no official tag; only
# results from the trusted verification sources carry release authority.
release-certification:
name: Release Certification
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$run_id" ]]; then
echo "No certified Build and Release run found for ${VERSION}; failing closed" >&2
exit 1
fi
if ! gh run download "$run_id" -n release-certification -D certification; then
echo "Build run ${run_id} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Input digest binding: every zip this image wraps must be the
# certified byte-for-byte content, never a mutable latest pointer.
while IFS=$'\t' read -r name location expected; do
echo "Verifying certified input ${name}"
actual="$(curl -fsSL "$location" | sha256sum | awk '{print $1}')"
if [[ "$actual" != "$expected" ]]; then
echo "Certified input ${name} does not match its recorded digest" >&2
exit 1
fi
done < <(jq -r '.deliverables[] | select(.name | contains("linux")) | [.name, .location, .sha256] | @tsv' certification/candidate.json)
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: [ build-check, release-certification ]
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 60
@@ -505,14 +443,6 @@ jobs:
# Output tags
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
# Staging identity for this run's promotion: official tags are only
# created after the staged image verifies against the certification,
# and they point at this same digest (no rebuild-after-verify).
STAGING_TAG="staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${VARIANT_SUFFIX}"
STAGING_TAGS="${{ env.REGISTRY_DOCKERHUB }}:${STAGING_TAG},${{ env.REGISTRY_GHCR }}:${STAGING_TAG},${{ env.REGISTRY_QUAY }}:${STAGING_TAG}"
echo "staging_tags=$STAGING_TAGS" >> "$GITHUB_OUTPUT"
echo "staging_tag=$STAGING_TAG" >> "$GITHUB_OUTPUT"
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
@@ -537,9 +467,7 @@ jobs:
file: ${{ matrix.file }}
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
# Push to staging tags only: official tags are created by the
# certification-gated promotion step below from this same digest.
tags: ${{ steps.meta.outputs.staging_tags }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
@@ -565,114 +493,6 @@ jobs:
no-cache: false
pull: true
# A later-built wrapper: the staged image must complete this platform's
# own binary identity verification before any official tag is written.
# It may not inherit an unverified package's conclusion.
- name: Verify staged image against certification
if: needs.build-check.outputs.should_push == 'true'
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
case "${{ matrix.variant }}" in
musl) libc="musl" ;;
glibc) libc="gnu" ;;
esac
for platform in linux/amd64 linux/arm64; do
case "$platform" in
linux/amd64) arch_sub="x86_64" ;;
linux/arm64) arch_sub="aarch64" ;;
esac
docker pull --platform "$platform" "$STAGING_REF"
container="$(docker create --platform "$platform" "$STAGING_REF")"
docker cp "$container:/usr/bin/rustfs" "$RUNNER_TEMP/staged-rustfs"
docker rm "$container" >/dev/null
actual="$(sha256sum "$RUNNER_TEMP/staged-rustfs" | awk '{print $1}')"
expected="$(jq -r --arg needle "linux-${arch_sub}-${libc}" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" || "$actual" != "$expected" ]]; then
echo "Staged image binary for ${platform} does not match the certified binary digest" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/staged-rustfs" \
--artifact-role binary
done
- name: Promote staged image to official tags
if: needs.build-check.outputs.should_push == 'true'
shell: bash
run: |
set -euo pipefail
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
IFS=',' read -r -a official_tags <<< "${{ steps.meta.outputs.tags }}"
for tag in "${official_tags[@]}"; do
docker buildx imagetools create -t "$tag" "$STAGING_REF"
echo "Promoted $tag -> $STAGING_REF"
done
# Downstream consumers (Helm) verify this record instead of inheriting
# an unverified wrapper's conclusion.
- name: Record image promotion
if: needs.build-check.outputs.should_push == 'true'
shell: bash
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
digest="$(docker buildx imagetools inspect "$STAGING_REF" --format '{{json .Manifest}}' | jq -r '.digest')"
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
jq -n \
--arg version "$VERSION" \
--arg digest "$digest" \
--arg staging_ref "$STAGING_REF" \
--arg candidate_sha256 "$candidate_sha" \
--arg tags "${{ steps.meta.outputs.tags }}" \
--slurpfile candidate certification/candidate.json \
'{schema_version: 1, version: $version, image_digest: $digest, staging_ref: $staging_ref,
promoted_tags: ($tags | split(",")), candidate_sha256: $candidate_sha256,
binary_sha256s: ($candidate[0].deliverables | map(.binary_sha256) | unique)}' \
> image-promotion-record.json
- name: Upload image promotion record
if: needs.build-check.outputs.should_push == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: image-promotion-record${{ matrix.suffix }}
path: image-promotion-record.json
if-no-files-found: error
retention-days: 30
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
-93
View File
@@ -30,102 +30,9 @@ permissions:
contents: read
jobs:
# Consume the same trusted release certification as every other publication
# exit and verify the image promotion record of the Docker workflow. The
# chart wraps no binary of its own, so it may only reference an image whose
# platform verification already passed; it must not inherit an unverified
# package's conclusion. A run without that evidence fails closed here and
# never reaches the chart repository.
release-certification:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
outputs:
version: ${{ steps.version.outputs.raw_tag }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -euo pipefail
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*) RAW_TAG="${RAW#refs/tags/}" ;;
*) RAW_TAG="$RAW" ;;
esac
echo "raw_tag=$RAW_TAG" >> "$GITHUB_OUTPUT"
- name: Download and verify release certification and image record
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
RAW_TAG: ${{ steps.version.outputs.raw_tag }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification image-record
build_run="$TRIGGERING_RUN_ID"
if [[ -z "$build_run" ]]; then
build_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$build_run" ]]; then
echo "No certified Build and Release run found for ${RAW_TAG}; failing closed" >&2
exit 1
fi
if ! gh run download "$build_run" -n release-certification -D certification; then
echo "Build run ${build_run} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
docker_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/docker.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
if [[ -z "$docker_run" ]] || ! gh run download "$docker_run" -n image-promotion-record -D image-record; then
echo "No image promotion record for ${RAW_TAG}; a chart may not wrap an unverified image" >&2
exit 1
fi
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
record_sha="$(jq -r '.candidate_sha256 // empty' image-record/image-promotion-record.json)"
if [[ "$record_sha" != "$candidate_sha" ]]; then
echo "Image promotion record does not belong to this certification" >&2
exit 1
fi
overlap="$(jq -r --slurpfile candidate certification/candidate.json \
'[.binary_sha256s[]] - ($candidate[0].deliverables | map(.binary_sha256)) | length' \
image-record/image-promotion-record.json)"
if [[ "$overlap" -ne 0 ]]; then
echo "Image promotion record carries binary digests outside the certification" >&2
exit 1
fi
build-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ release-certification ]
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
+14 -210
View File
@@ -225,57 +225,10 @@ jobs:
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Consume the trusted release certification of the resolved build run and
# verify the certified input digests before any package is wrapped. A run
# without that certification fails closed here; packages built later must
# complete their own verification and cannot inherit an unverified
# conclusion.
release-certification:
name: Release Certification
needs: resolve
if: needs.resolve.result == 'success'
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
if [[ "$BUILD_TYPE" == "development" ]]; then
# Development artifacts are explicitly uncertified and never touch
# official or latest channels; only the staging/dev paths remain.
jq -n '{schema_version: 1, certified: false, reason: "development artifacts are explicitly uncertified"}' \
> certification/development-uncertified.json
exit 0
fi
if ! gh run download "$BUILD_RUN_ID" -n release-certification -D certification; then
echo "Build run ${BUILD_RUN_ID} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Build DEB and RPM packages for each architecture and libc variant
package:
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: [ resolve, release-certification ]
needs: resolve
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
strategy:
@@ -541,55 +494,6 @@ jobs:
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Download certification evidence
if: needs.resolve.outputs.build_type != 'development'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
# Later-built wrapper verification: the binary inside each package must
# match the certified binary digest for this platform before anything is
# uploaded, and the package itself stays on staging until promotion.
- name: Verify wrapped binary against certification
if: needs.resolve.outputs.build_type != 'development'
shell: bash
env:
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
ARTIFACT_NAME: ${{ matrix.artifact_name }}
run: |
set -euo pipefail
expected="$(jq -r --arg needle "$ARTIFACT_NAME" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" ]]; then
echo "No certified binary digest recorded for ${ARTIFACT_NAME}" >&2
exit 1
fi
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y cpio
fi
for package_file in "$DEB_FILE" "$RPM_FILE"; do
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
done
- name: Upload packages to artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
@@ -627,14 +531,10 @@ jobs:
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Build-phase writes stay on staging for release types: the official
# packages/release and packages/latest paths are written only by the
# certification-gated promote-packages job, from these same staged
# objects. Development packages keep the uncertified dev channel.
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}"
R2_PREFIX="artifacts/rustfs/packages/release"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
@@ -649,6 +549,18 @@ jobs:
echo "✅ Upload complete"
# Also upload as latest for release/prerelease
if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading latest: $(basename "$f")"
aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Latest packages updated"
fi
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
env:
@@ -702,114 +614,6 @@ jobs:
echo "✅ GitHub release assets updated"
# The only writer of the official packages/release and packages/latest R2
# paths. Promotion re-checks each package's wrapped binary against the
# passed certification list and copies the staged objects byte-for-byte:
# no rebuild-after-verify, no mutable latest resolution downstream.
promote-packages:
name: Promote Packages
needs: [ resolve, package, release-certification ]
# Target-channel lock for the official package paths.
concurrency:
group: release-channel-r2-packages
cancel-in-progress: false
if: needs.resolve.outputs.tag != '' && (needs.resolve.outputs.build_type == 'release' || needs.resolve.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download packages and certification
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./packages
pattern: packages-*
merge-multiple: true
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify and promote staged packages
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "❌ R2 credentials missing; refusing to promote to the official channel" >&2
exit 1
fi
STAGING_PREFIX="s3://${R2_BUCKET}/artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
RELEASE_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/release/"
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get install -y cpio
fi
promoted=0
for package_file in packages/*.deb packages/*.rpm; do
[[ -f "$package_file" ]] || continue
name="$(basename "$package_file")"
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
staged="$(mktemp)"
aws s3 cp "${STAGING_PREFIX}${name}" "$staged" --endpoint-url "$R2_ENDPOINT" --only-show-errors
staged_sha="$(sha256sum "$staged" | awk '{print $1}')"
built_sha="$(sha256sum "$package_file" | awk '{print $1}')"
if [[ "$staged_sha" != "$built_sha" ]]; then
echo "❌ Staged package ${name} does not match the verified bytes" >&2
exit 1
fi
aws s3 cp "$staged" "${RELEASE_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
aws s3 cp "$staged" "${LATEST_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
rm -f "$staged"
promoted=$((promoted + 1))
echo "✅ Promoted ${name} (${built_sha})"
done
if [[ "$promoted" -eq 0 ]]; then
echo "No staged packages found to promote" >&2
exit 1
fi
# Summary
summary:
name: Summary
@@ -1,421 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: RustFS Validation Sweep
# Executes the migrated canonical validation cases (rustfs/auto-testing#113)
# on the shared functional fleet. Everything with a lane executor is swept in
# rotating nightly batches so the whole set is covered roughly weekly; a
# single case or one module can be dispatched manually. That is the
# implemented catalog (66 today) plus the eight planned cases whose dedicated
# executors already exist and have live PASS evidence - fresh nightly
# evidence feeds their acceptance review. heal.* cases stay on the
# interactive validation flow: they require reviewed native/tool collectors
# built by separate validation-tools runs before fleet execution.
on:
schedule:
# After the nightly functional chain settles (chain starts ~16:48 UTC and
# runs for about two hours).
- cron: "30 19 * * *"
workflow_dispatch:
inputs:
case:
description: 'Single canonical validation case id (overrides module and rotation), e.g. pool.decommission'
type: string
required: false
default: ''
module:
description: 'Restrict the sweep to one catalog module, e.g. bucket-replication'
type: string
required: false
default: ''
limit:
description: 'Maximum cases per sweep (0 = run everything selected)'
type: number
required: false
default: 12
build_run_id:
description: 'Successful nightly package run; empty resolves the latest'
type: string
required: false
default: ''
build_run_attempt:
description: 'Attempt of that package run'
type: string
required: false
default: '1'
permissions:
contents: read
concurrency:
group: rustfs-validation-sweep
cancel-in-progress: false
env:
RUSTFS_NODES: ${{ secrets.RUSTFS_NODES || vars.RUSTFS_NODES || 'vm000 vm001 vm002 vm003' }}
RUSTFS_SSH_USER: ${{ secrets.RUSTFS_SSH_USER || vars.RUSTFS_SSH_USER || 'azureuser' }}
RUSTFS_ACCESS_KEY: ${{ secrets.RUSTFS_ACCESS_KEY }}
RUSTFS_SECRET_KEY: ${{ secrets.RUSTFS_SECRET_KEY }}
RUSTFS_NIGHTLY_PACKAGE_URL: ${{ vars.RUSTFS_NIGHTLY_PACKAGE_URL || 'https://dl.rustfs.com/artifacts/rustfs/packages/nightly/rustfs-nightly-latest.deb' }}
PF_TESTING_GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
jobs:
sweep:
if: github.repository == 'rustfs/rustfs'
runs-on: smoke-testing
timeout-minutes: 150
steps:
- name: Checkout chain tooling
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Resolve nightly candidate
id: candidate
env:
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
BUILD_RUN_ID: ${{ inputs.build_run_id }}
BUILD_RUN_ATTEMPT: ${{ inputs.build_run_attempt }}
CHAIN_OUTPUT: ${{ runner.temp }}/validation-sweep-candidate.json
run: |
set -euo pipefail
python3 scripts/resolve_functional_candidate.py
python3 - <<'PY'
import json
import os
from pathlib import Path
root = Path(os.environ["RUNNER_TEMP"]) / "validation-sweep"
root.mkdir(parents=True, exist_ok=True)
candidate = json.loads((Path(os.environ["RUNNER_TEMP"]) / "validation-sweep-candidate.json").read_text())
manifest = candidate["manifest"]
(root / "candidate.json").write_text(json.dumps({
"package_url": manifest["package_url"],
"package_sha256": manifest["package_sha256"],
"source_sha": manifest["source_sha"],
}, sort_keys=True) + "\n")
with open(os.environ["GITHUB_OUTPUT"], "a") as output:
output.write("testing_sha=" + candidate["testing_sha"] + "\n")
output.write("package_url=" + manifest["package_url"] + "\n")
output.write("source_sha=" + manifest["source_sha"] + "\n")
PY
- name: Checkout auto-testing scripts
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
repository: rustfs/auto-testing
ref: ${{ steps.candidate.outputs.testing_sha }}
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
path: auto-testing
persist-credentials: false
- name: Select sweep cases
id: select
env:
SELECTED_CASE: ${{ inputs.case }}
SELECTED_MODULE: ${{ inputs.module }}
CASE_LIMIT: ${{ inputs.limit }}
SWEEP_EVENT: ${{ github.event_name }}
SWEEP_ROOT: ${{ runner.temp }}/validation-sweep
run: |
set -euo pipefail
python3 - <<'PY'
import datetime
import json
import os
from pathlib import Path
sweep_root = Path(os.environ["SWEEP_ROOT"])
catalog = json.loads(Path("auto-testing/validation-cases.json").read_text())
def sweepable(case):
status = case.get("source_status")
if status == "implemented":
return True
# The remaining planned non-heal cases route to the lane
# executors and already have live PASS evidence; sweeping them
# nightly produces fresh evidence for their acceptance review.
return status == "planned" and not case["id"].startswith("heal.")
selected = [(case["id"], module["id"]) for module in catalog["modules"]
for case in module["cases"]
if sweepable(case)]
if not selected:
raise SystemExit("catalog has no sweepable cases")
one_case = os.environ.get("SELECTED_CASE", "").strip()
module_filter = os.environ.get("SELECTED_MODULE", "").strip()
known = {case_id for case_id, _ in selected}
modules = {module_id for _, module_id in selected}
if one_case:
if one_case not in known:
raise SystemExit(f"unknown case or no lane executor for it: {one_case}")
selected = [(one_case, dict(selected)[one_case])]
elif module_filter:
if module_filter not in modules:
raise SystemExit(f"unknown module or no implemented cases: {module_filter}")
selected = [pair for pair in selected if pair[1] == module_filter]
elif os.environ.get("SWEEP_EVENT") == "schedule":
# Stateless rotation: every night take one window of the catalog
# so a full sweep needs len(selected)/limit nights and no state
# has to be carried between runs.
limit = max(1, int(os.environ.get("CASE_LIMIT") or 12))
day = datetime.datetime.now(datetime.timezone.utc).timetuple().tm_yday
start = (day * limit) % len(selected)
selected = (selected[start:] + selected[:start])[:limit]
limit = int(os.environ.get("CASE_LIMIT") or 0)
if one_case or module_filter or (not os.environ.get("SWEEP_EVENT") == "schedule"):
if limit > 0:
selected = selected[:limit]
(sweep_root / "sweep-cases.txt").write_text("".join(f"{case_id}\n" for case_id, _ in selected))
with open(os.environ["GITHUB_OUTPUT"], "a") as output:
output.write(f"count={len(selected)}\n")
print(f"selected {len(selected)} case(s):", ", ".join(c for c, _ in selected))
PY
- name: Admit clean shared environment
id: environment
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-validation-sweep.json
run: python3 scripts/functional_environment.py enter --suite heal
- name: Run validation cases
id: run_cases
timeout-minutes: 120
env:
AUTO_TESTING_SHA: ${{ steps.candidate.outputs.testing_sha }}
SWEEP_ROOT: ${{ runner.temp }}/validation-sweep
run: |
set -uo pipefail
mkdir -p "${SWEEP_ROOT}/results"
: > "${SWEEP_ROOT}/results.txt"
while read -r CASE_ID; do
[ -n "${CASE_ID}" ] || continue
echo "::group::case ${CASE_ID}"
rm -rf "${SWEEP_ROOT}/run"
if ./auto-testing/scripts/run_validation_case.sh \
"${CASE_ID}" "${SWEEP_ROOT}/candidate.json" \
"${AUTO_TESTING_SHA}" "${SWEEP_ROOT}/run" \
> "${SWEEP_ROOT}/results/${CASE_ID//\//_}.log" 2>&1; then
printf '%s\tPASS\n' "${CASE_ID}" >> "${SWEEP_ROOT}/results.txt"
echo "case ${CASE_ID}: PASS"
else
printf '%s\tFAIL\n' "${CASE_ID}" >> "${SWEEP_ROOT}/results.txt"
echo "case ${CASE_ID}: FAIL (log retained in the sweep artifact)"
fi
if [ -f "${SWEEP_ROOT}/run/report.md" ]; then
cp "${SWEEP_ROOT}/run/report.md" "${SWEEP_ROOT}/results/${CASE_ID//\//_}.report.md" 2>/dev/null || true
fi
echo "::endgroup::"
# Fleet hygiene between cases: the case runners set up their own
# topology, this purge guarantees the next case starts clean even
# when the previous one died mid-run.
for node in ${RUSTFS_NODES}; do
ssh -o BatchMode=yes -o ConnectTimeout=10 "${RUSTFS_SSH_USER}@${node}" '
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
${SUDO} systemctl stop rustfs 2>/dev/null || true
if ${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
${SUDO} dpkg -P rustfs
fi
for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
${SUDO} rm -f /etc/default/rustfs
' || echo "inter-case cleanup failed on ${node} (continuing)"
done
done < "${SWEEP_ROOT}/sweep-cases.txt"
echo "sweep finished: $(wc -l < "${SWEEP_ROOT}/results.txt") case(s) executed"
- name: Generate sweep report
if: ${{ always() && steps.environment.outcome == 'success' }}
env:
SWEEP_ROOT: ${{ runner.temp }}/validation-sweep
PACKAGE_URL: ${{ steps.candidate.outputs.package_url }}
SOURCE_SHA: ${{ steps.candidate.outputs.source_sha }}
run: |
set -euo pipefail
python3 - <<'PY'
import json
import os
from pathlib import Path
root = Path(os.environ["SWEEP_ROOT"])
catalog = json.loads(Path("auto-testing/validation-cases.json").read_text())
module_of = {case["id"]: module["id"] for module in catalog["modules"] for case in module["cases"]}
rows = []
for line in (root / "results.txt").read_text().splitlines():
if not line:
continue
case_id, status = line.split("\t")
rows.append((case_id, module_of.get(case_id, "?"), status))
passed = sum(1 for _, _, s in rows if s == "PASS")
failed = len(rows) - passed
run_url = (f"{os.environ.get('GITHUB_SERVER_URL', 'https://github.com')}/"
f"{os.environ.get('GITHUB_REPOSITORY', 'rustfs/rustfs')}/actions/runs/"
f"{os.environ.get('GITHUB_RUN_ID', '')}")
report = [
"# RustFS validation sweep report", "",
f"- Run: {run_url}",
f"- Attempt: {os.environ.get('GITHUB_RUN_ATTEMPT', '')}",
f"- Trigger: {os.environ.get('GITHUB_EVENT_NAME', '')}",
f"- Package: {os.environ.get('PACKAGE_URL', 'nightly (R2 latest)')}",
f"- Source SHA: {os.environ.get('SOURCE_SHA', '')}",
f"- auto-testing: {os.environ.get('AUTO_TESTING_SHA', '')}", "",
"Canonical validation cases (rustfs/auto-testing#113) executed by the sweep, grouped by catalog module.", "",
]
# Group the executed cases by catalog module (the major category:
# pool lifecycle, heal, ILM expiration, bucket/site replication, kms,
# data services, identity, s3 compatibility). Per-module subtotals
# deliberately use "- Subtotal:" so the dashboard overview regexes
# (which anchor on "- Total:") keep reading the grand totals below.
by_module = {}
module_order = []
for case_id, module, status in rows:
if module not in by_module:
by_module[module] = []
module_order.append(module)
by_module[module].append((case_id, status))
report += ["## Case Summary", ""]
for module in module_order:
cases = by_module[module]
mod_pass = sum(1 for _, st in cases if st == "PASS")
mod_fail = len(cases) - mod_pass
report += [f"### {module} ({len(cases)})", "",
"| Case | Status |", "| --- | --- |"]
report += [f"| {case_id} | {status} |" for case_id, status in cases]
report += ["", f"- Subtotal: {mod_pass} passed, {mod_fail} failed", ""]
report += [
"## Totals", "",
f"- Total: {len(rows)}",
f"- PASS: {passed}",
f"- FAIL: {failed}",
"- UNSUPPORTED: 0", "",
"Failing cases keep this workflow red only as a report; the sweep artifact",
"retains every case log and per-case report for diagnosis.", "",
]
(root / "report.md").write_text("\n".join(report))
(root / "cases.md").write_text("\n".join(f"| {c} | {m} | {s} |" for c, m, s in rows) + "\n")
PY
cat "${SWEEP_ROOT}/report.md" >> "${GITHUB_STEP_SUMMARY}"
- name: Upload functional report to dashboard
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 2
continue-on-error: true
env:
GH_TOKEN: ${{ env.PF_TESTING_GH_TOKEN }}
REPORT_FILE: ${{ runner.temp }}/validation-sweep/report.md
SUITE: validation
run: |
set -euo pipefail
if [ -z "${GH_TOKEN:-}" ]; then
echo "PF_TESTING_GH_TOKEN is not configured; skipping dashboard upload"
exit 0
fi
DATE="$(date -u +%Y-%m-%d)"
REPORT_PATH="functional-reports/${SUITE}/${DATE}.md"
B64_FILE="$(mktemp)"
python3 -c 'import base64,sys;print(base64.b64encode(open(sys.argv[1],"rb").read()).decode())' "${REPORT_FILE}" > "${B64_FILE}"
SHA="$(gh api "repos/rustfs/dashboard/contents/${REPORT_PATH}" -q '.sha' 2>/dev/null || true)"
if [ -n "${SHA}" ]; then
jq -n --arg msg "report(${SUITE}): ${DATE}" --rawfile content "${B64_FILE}" --arg sha "${SHA}" \
'{message:$msg, content:($content|rtrimstr("\n")), sha:$sha}' \
| gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
else
jq -n --arg msg "report(${SUITE}): ${DATE}" --rawfile content "${B64_FILE}" \
'{message:$msg, content:($content|rtrimstr("\n"))}' \
| gh api --method PUT "repos/rustfs/dashboard/contents/${REPORT_PATH}" --input - >/dev/null
fi
echo "dashboard updated: ${REPORT_PATH}"
- name: Retain sweep evidence
if: ${{ always() }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: rustfs-validation-sweep-${{ github.run_id }}-${{ github.run_attempt }}
path: |
${{ runner.temp }}/validation-sweep/report.md
${{ runner.temp }}/validation-sweep/results.txt
${{ runner.temp }}/validation-sweep/sweep-cases.txt
${{ runner.temp }}/validation-sweep/candidate.json
${{ runner.temp }}/validation-sweep/results/
if-no-files-found: error
retention-days: 30
- name: Cleanup environment (after)
id: environment_cleanup
timeout-minutes: 5
if: ${{ always() && steps.environment.outcome == 'success' }}
run: |
set -uo pipefail
cleanup_status=0
for node in ${RUSTFS_NODES}; do
ssh -o BatchMode=yes -o ConnectTimeout=10 "${RUSTFS_SSH_USER}@${node}" '
set -euo pipefail
SUDO=""; [ "$(id -u)" -ne 0 ] && SUDO="sudo -n"
${SUDO} systemctl stop rustfs 2>/dev/null || true
if ${SUDO} dpkg -l rustfs 2>/dev/null | grep -q "^ii"; then
${SUDO} dpkg -P rustfs
fi
for i in 1 2 3 4; do ${SUDO} rm -rf /data/rustfs${i}/mnmd; done
${SUDO} rm -f /etc/default/rustfs
' || cleanup_status=$?
done
exit "${cleanup_status}"
- name: Release environment only after successful cleanup
if: ${{ always() && steps.environment.outcome == 'success' }}
timeout-minutes: 5
env:
FUNCTIONAL_ENVIRONMENT_RECEIPT: ${{ runner.temp }}/environment-validation-sweep.json
CLEANUP_OUTCOME: ${{ steps.environment_cleanup.outcome }}
run: python3 scripts/functional_environment.py finish --suite heal --cleanup-outcome "$CLEANUP_OUTCOME"
- name: Notify on failure
if: failure()
run: echo "RustFS validation sweep failed (harness/environment breakdown); see the sweep artifact for case logs."
- name: Enforce sweep result
if: ${{ always() && steps.environment.outcome == 'success' }}
env:
SWEEP_ROOT: ${{ runner.temp }}/validation-sweep
run: |
set -euo pipefail
if grep -q $'\tFAIL$' "${SWEEP_ROOT}/results.txt"; then
echo "validation sweep has failing cases:"
grep $'\tFAIL$' "${SWEEP_ROOT}/results.txt"
exit 1
fi
alert-on-failure:
name: Alert on scheduled failure
# A scheduled sweep failure usually means a canonical validation case
# regressed or the fleet environment broke; surface it as a tracking
# issue instead of leaving it in the Actions tab.
needs: [sweep]
if: github.repository == 'rustfs/rustfs' && (always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure'))
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v5.0.0
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}
+1 -1
View File
@@ -155,7 +155,7 @@ default build (lifecycle:
- ⚠️ PARTIALLY VIOLATED: serving-side `s3s` references remain in ecstore
(bucket metadata/replication/lifecycle DTOs and error mapping). The
count is ratcheted shrink-only by `scripts/check_s3s_footprint.sh`
(`S3S_ECSTORE_FILES_BASELINE`; the `object_lock` module was converted to
(the `ecstore_files` line of `.config/s3s-footprint-baseline.txt`; the `object_lock` module was converted to
storage-level types as the first ratchet step). Target state: the
baseline reaches zero and ecstore's `Cargo.toml` drops `s3s`.
Generated
+123 -63
View File
@@ -1321,9 +1321,9 @@ dependencies = [
[[package]]
name = "aws-smithy-schema"
version = "1.1.2"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9543fc722ed4865ab29fb66bd7ed0bef7b029d265ba8c587144511e30006d431"
checksum = "bec8dae1015a6d959a5faf86fbdf77486cfe19e4ccb6ee009d8a46efc45028a4"
dependencies = [
"aws-smithy-runtime-api",
"aws-smithy-types",
@@ -4198,9 +4198,9 @@ checksum = "34aa73646ffb006b8f5147f3dc182bd4bcb190227ce861fc4a4844bf8e3cb2c0"
[[package]]
name = "encoding_rs"
version = "0.8.44"
version = "0.8.42"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "89253175e704ab869b04f9db224d38630e0211a53841a79181deae0889e169f5"
checksum = "8e985e0451871ad22fb8d2b6b076e2028a502a0d3950998c2c5c0a4f9b5d9679"
dependencies = [
"cfg-if",
"core_detect",
@@ -5277,9 +5277,9 @@ dependencies = [
[[package]]
name = "hotpath"
version = "0.30.2"
version = "0.30.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b39e941010d86063ff233bf8325269bf68097fb24ba67000fbddc0a410e8056e"
checksum = "502dc37a7c218b9b1d3a0ccfb904da81d10d6dda0e899c722c503092d55576ea"
dependencies = [
"arc-swap",
"async-channel",
@@ -5312,15 +5312,15 @@ dependencies = [
[[package]]
name = "hotpath-drain"
version = "0.30.2"
version = "0.30.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d0faae1a9e1fd8ce5debaaaf452dcbe833fc8a8a4f3779d2ec8828b116c3c16a"
checksum = "0ea17d04c231b90b1f68a91f6f80fb0762c5dea07c611dce9ae157b56a50ff4c"
[[package]]
name = "hotpath-macros"
version = "0.30.2"
version = "0.30.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a24f5179ff680132003306354efd0ac8cde12e87b1a788719c836e91af35cb58"
checksum = "a76e0af911930fc97abc766b393b48cdca77fea9f5f6b388cf2123a6d9c30deb"
dependencies = [
"proc-macro2",
"quote",
@@ -5899,9 +5899,9 @@ dependencies = [
[[package]]
name = "js-sys"
version = "0.3.107"
version = "0.3.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8dc09d2154098bbf4ef54eab6b117a3c24cbbffe94046b16ee6d6d4510584d3a"
checksum = "7883d941dae510fb2d978fc3fe018c71c9e2892fd38854de3e8b92c2e5ad9cc5"
dependencies = [
"cfg-if",
"futures-util",
@@ -6094,9 +6094,9 @@ dependencies = [
[[package]]
name = "libbz2-rs-sys"
version = "0.2.6"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32f454fd6831a29c0e827ed075bca2a15be201bd57f4168f0fd1041124eab17e"
checksum = "34b357333733e8260735ba5894eb928c02ecc69c78715f01a8019e7fa7f2db4c"
[[package]]
name = "libc"
@@ -8305,9 +8305,9 @@ dependencies = [
[[package]]
name = "pulldown-cmark"
version = "0.13.6"
version = "0.13.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3db2be46835f013dd3f4fa7f5f7402c537787b54ebb6fe2b94bbdd7e2669bb7b"
checksum = "278e9268c93b9e498a2da46c34d90ba93a3255eb99009646398f34bc73ab36eb"
dependencies = [
"bitflags 2.13.2",
"memchr",
@@ -8520,6 +8520,15 @@ version = "1.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
[[package]]
name = "quick-xml"
version = "0.41.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e660451e55124f798a69a5af3f49ccfbefbd41910eefd25caf2393e1f3473ec1"
dependencies = [
"memchr",
]
[[package]]
name = "quick-xml"
version = "0.42.0"
@@ -8823,9 +8832,9 @@ dependencies = [
[[package]]
name = "redis"
version = "1.8.1"
version = "1.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "db7f3d75b5a3301746bff39a767bf4c5c42506b08513527308b3875bfb64a031"
checksum = "7acd809e80afa754c63f586955699d8990efe1c4b6613183ee5f027a11fbc8b5"
dependencies = [
"arc-swap",
"arcstr",
@@ -9384,7 +9393,7 @@ dependencies = [
"proptest",
"prost 0.14.4",
"pyroscope",
"quick-xml",
"quick-xml 0.42.0",
"quinn",
"rand 0.10.3",
"rcgen",
@@ -9403,6 +9412,7 @@ dependencies = [
"rustfs-ecstore",
"rustfs-extension-schema",
"rustfs-filemeta",
"rustfs-gateway-types",
"rustfs-heal",
"rustfs-heal-contracts",
"rustfs-iam",
@@ -9681,6 +9691,7 @@ dependencies = [
"rustfs-data-usage",
"rustfs-erasure-codec",
"rustfs-filemeta",
"rustfs-gateway-types",
"rustfs-heal-contracts",
"rustfs-io-metrics",
"rustfs-lifecycle",
@@ -9786,6 +9797,48 @@ dependencies = [
"xxhash-rust",
]
[[package]]
name = "rustfs-gateway-stream"
version = "0.2.0"
source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
dependencies = [
"bitflags 2.13.2",
"bytes",
"http 1.5.0",
"http-body",
]
[[package]]
name = "rustfs-gateway-types"
version = "0.32.0+aws.2026-10-02"
source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
dependencies = [
"bytes",
"crc-fast",
"futures-core",
"hex",
"http 1.5.0",
"md-5 0.11.0",
"percent-encoding",
"rustfs-gateway-stream",
"rustfs-gateway-xml",
"s3s",
"sha1 0.11.0",
"sha2 0.11.1",
"time",
"twox-hash",
"unicode-normalization",
"zeroize",
]
[[package]]
name = "rustfs-gateway-xml"
version = "0.4.0"
source = "git+https://github.com/rustfs/gateway.git?rev=21283e874c2381bab611e9347acbc5b92e9e2b3c#21283e874c2381bab611e9347acbc5b92e9e2b3c"
dependencies = [
"quick-xml 0.41.0",
]
[[package]]
name = "rustfs-heal"
version = "1.0.2"
@@ -10056,10 +10109,10 @@ dependencies = [
"metrics-util",
"proptest",
"rustfs-config",
"rustfs-gateway-types",
"rustfs-replication",
"rustfs-scanner-metrics",
"rustfs-storage-api",
"s3s",
"serial_test",
"temp-env",
"time",
@@ -10122,7 +10175,6 @@ dependencies = [
"reqwest",
"rmp-serde",
"rustfs-signer",
"s3s",
"serde",
"serde_json",
"sysinfo",
@@ -10162,7 +10214,7 @@ dependencies = [
"jiff",
"metrics",
"percent-encoding",
"quick-xml",
"quick-xml 0.42.0",
"rayon",
"rustc-hash",
"rustfs-config",
@@ -10343,10 +10395,13 @@ dependencies = [
"rustfs-config",
"rustfs-credentials",
"rustfs-ecstore",
"rustfs-gateway-stream",
"rustfs-gateway-types",
"rustfs-iam",
"rustfs-keystone",
"rustfs-policy",
"rustfs-rio",
"rustfs-s3-types",
"rustfs-storage-api",
"rustfs-test-utils",
"rustfs-tls-runtime",
@@ -10354,13 +10409,13 @@ dependencies = [
"rustfs-trusted-proxies",
"rustfs-utils",
"rustls",
"s3s",
"serde",
"serde_json",
"sha1 0.11.0",
"sha2 0.11.1",
"socket2",
"subtle",
"sync_wrapper",
"tempfile",
"thiserror 2.0.21",
"time",
@@ -10410,7 +10465,7 @@ dependencies = [
"regex",
"rmp",
"rmp-serde",
"s3s",
"rustfs-gateway-types",
"serde",
"time",
"tokio",
@@ -10505,11 +10560,13 @@ dependencies = [
"hyper-util",
"lazy_static",
"md-5 0.11.0",
"quick-xml",
"quick-xml 0.42.0",
"rand 0.10.3",
"rustfs-checksums",
"rustfs-config",
"rustfs-gateway-types",
"rustfs-rio",
"rustfs-s3-types",
"rustfs-signer",
"rustfs-storage-api",
"rustfs-tls-runtime",
@@ -10543,9 +10600,19 @@ dependencies = [
name = "rustfs-s3-types"
version = "1.0.2"
dependencies = [
"bytes",
"futures-core",
"hotpath",
"http 1.5.0",
"http-body",
"http-body-util",
"hyper",
"hyper-util",
"pin-project-lite",
"s3s",
"serde",
"serde_json",
"tokio",
]
[[package]]
@@ -10569,9 +10636,10 @@ dependencies = [
"parking_lot 0.13.0",
"rustfs-common",
"rustfs-ecstore",
"rustfs-gateway-types",
"rustfs-s3-types",
"rustfs-storage-api",
"rustfs-test-utils",
"s3s",
"serde_json",
"serial_test",
"thiserror 2.0.21",
@@ -10597,7 +10665,6 @@ dependencies = [
"parking_lot 0.13.0",
"rustfs-s3select-api",
"rustfs-test-utils",
"s3s",
"serde",
"serde_json",
"tokio",
@@ -10629,6 +10696,7 @@ dependencies = [
"rustfs-data-usage",
"rustfs-ecstore",
"rustfs-filemeta",
"rustfs-gateway-types",
"rustfs-heal",
"rustfs-heal-contracts",
"rustfs-lifecycle",
@@ -10686,9 +10754,10 @@ dependencies = [
"bytes",
"hotpath",
"http 1.5.0",
"http-body",
"hyper",
"rustfs-s3-types",
"rustfs-utils",
"s3s",
"serde_urlencoded",
"thiserror 2.0.21",
"time",
@@ -10746,7 +10815,6 @@ dependencies = [
"rustfs-utils",
"rustls",
"rustls-native-certs",
"s3s",
"serde",
"serde_json",
"sha2 0.11.1",
@@ -11137,7 +11205,7 @@ dependencies = [
"numeric_cast",
"phf 0.14.0",
"pin-project-lite",
"quick-xml",
"quick-xml 0.42.0",
"regex",
"s3s-multipart",
"s3s-rfc2047",
@@ -12217,9 +12285,9 @@ dependencies = [
[[package]]
name = "sysinfo"
version = "0.40.1"
version = "0.40.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0e8a3edbe0394ce115f3917a00349dd242be09eb5a091ed509eaf67b55cfb8a8"
checksum = "110e37d98880fd804cd47509b90a8444f7aaaab655ae625e83cd442a8bc2ebf2"
dependencies = [
"libc",
"memchr",
@@ -12710,9 +12778,9 @@ checksum = "06bdbd8cfc056b8d2e2e85f29b56a3bdbecb527cef81eb39e3e7b98af4652770"
[[package]]
name = "tonic"
version = "0.14.9"
version = "0.14.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e2cdfc673aabd30fe8fee92e65837b644c37c5d47f0f97ae6d5e0f1237d15e1b"
checksum = "cd1fa9b1a51188f3266a823208078315811269d06d272c3a61be0b9a194194ff"
dependencies = [
"async-trait",
"axum",
@@ -12743,9 +12811,9 @@ dependencies = [
[[package]]
name = "tonic-build"
version = "0.14.9"
version = "0.14.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ac6d916d206ed7bdda1778ddff9bda43303789c258877e7bc22fb44ee5c984b4"
checksum = "2c07bd4e9f99e2a3ed8bebad596ae02a94e53745b68acc6900e22c3475c8bc44"
dependencies = [
"prettyplease 0.2.37",
"proc-macro2",
@@ -12755,9 +12823,9 @@ dependencies = [
[[package]]
name = "tonic-prost"
version = "0.14.9"
version = "0.14.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "eba464054e2226f182a34489b8b079bf4c6e3e7ab5fd7fc30e024f24aa92a484"
checksum = "f63c616ffaa648f7c5dd5657e97d9a285fcb44ff3b70da761d436f5a13ff7be6"
dependencies = [
"bytes",
"prost 0.14.4",
@@ -12766,9 +12834,9 @@ dependencies = [
[[package]]
name = "tonic-prost-build"
version = "0.14.9"
version = "0.14.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ea216ddc448e08e7839ab09e20e60b06b6a1c32745ccb212c0ebd161c199f1b"
checksum = "4a05d8957fac6b3542999796cb4449d45da790751962f974a457e5e00c43d78c"
dependencies = [
"prettyplease 0.2.37",
"proc-macro2",
@@ -13161,12 +13229,9 @@ checksum = "daf8dba3b7eb870caf1ddeed7bc9d2a049f3cfdfae7cb521b087cc33ae4c49da"
[[package]]
name = "utf8_iter"
version = "1.1.0"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af3bd9d3e571c26fcce9d033b87ea5ab061b8d2ffe9b7a5675d810d68f03edf0"
dependencies = [
"utf_types",
]
checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be"
[[package]]
name = "utf8parse"
@@ -13174,12 +13239,6 @@ version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "utf_types"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3a3b425286980a367278453f3b6d28cdcb134b574e5a35824a6ff5eebad50bb6"
[[package]]
name = "uuid"
version = "1.29.0"
@@ -13294,11 +13353,12 @@ dependencies = [
[[package]]
name = "wasm-bindgen"
version = "0.2.130"
version = "0.2.129"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a0181c14ddd6c7b8b0b0132a5f1ea2f32be0d893cd3ea97b6e3b28886b8a3d1c"
checksum = "9bb54f33acc68fd454578d9820b0bde1a1a3d17aa17bb7b6595806d02886d409"
dependencies = [
"cfg-if",
"once_cell",
"rustversion",
"wasm-bindgen-macro",
"wasm-bindgen-shared",
@@ -13306,9 +13366,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-futures"
version = "0.4.80"
version = "0.4.79"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5e14e0327cb5368ad60ccb22d04a9827a693c95a926d8d125e9fa39e0baec501"
checksum = "3cbab34de2d982e9b48e18d216d04c4a6f641066ff19ffb699980f591ee3610e"
dependencies = [
"js-sys",
"tokio",
@@ -13317,9 +13377,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-macro"
version = "0.2.130"
version = "0.2.129"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6e931a02a39abdbf66966fc35902b5ed58344750a70885bed4ebb3d8eaf28ed9"
checksum = "2e29d0c35b16e224a7eeb5cd2d25e3e1968fbd65604117b44d3b789d00ee8535"
dependencies = [
"quote",
"wasm-bindgen-macro-support",
@@ -13327,9 +13387,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-macro-support"
version = "0.2.130"
version = "0.2.129"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2ca4b54280e455bd831e33a6b8e734baf432fe63fbc35c42ac14ce67a043a53a"
checksum = "6f501a8bc3719dba86ef8ae4728879c08001bea749eb1333ac5b91e040e2a6b7"
dependencies = [
"bumpalo",
"proc-macro2",
@@ -13340,9 +13400,9 @@ dependencies = [
[[package]]
name = "wasm-bindgen-shared"
version = "0.2.130"
version = "0.2.129"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3b1f9c3798725016c6ee5f2f57869ec0831e8bbc0ab0310a86e787e97f41389e"
checksum = "23f0c9c52aa7cd7d77769a4cfe2a9adb1b331f489a41d912ce14513d5ab995c6"
dependencies = [
"unicode-ident",
]
@@ -13362,9 +13422,9 @@ dependencies = [
[[package]]
name = "web-sys"
version = "0.3.107"
version = "0.3.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7d179596ff5ec0f6de4db64d54011bde6289cc49603d3a6e4ce3f329207ab88"
checksum = "88261b9deccee56594c11a3460c462c41f58d148598fe70ad77070126a68aba4"
dependencies = [
"js-sys",
"wasm-bindgen",
+19 -7
View File
@@ -48,7 +48,7 @@ members = [
"crates/replication", # Replication contracts and wire formats
"crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
"crates/s3-client", # S3 client for engine-side consumption of remote S3 endpoints (tiering, transition targets)
"crates/s3-types", # S3 event type definitions
"crates/s3-types", # S3 event and error type definitions
"crates/s3-ops", # S3 operation definitions and mapping
"crates/s3select-api", # S3 Select API interface
"crates/s3select-query", # S3 Select query engine
@@ -143,6 +143,13 @@ rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
rustfs-utils = { path = "crates/utils", version = "1.0.2" }
rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
# The gateway's S3 types: the DTOs and persisted bucket-configuration codecs the
# engine crates move onto from s3s (rustfs/backlog#2745). The one pin of the
# gateway in this workspace; no crate may enable a `compat-s3s*` feature except
# the s3s-facing edge of the rustfs binary.
rustfs-gateway-types = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.32.0", default-features = false }
rustfs-gateway-stream = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.2.0", default-features = false }
# Async Runtime and Networking
async-channel = "2.5.0"
async_zip = { default-features = false, version = "0.0.19" }
@@ -175,9 +182,9 @@ tokio-rustls = { default-features = false, version = "0.26.6" }
tokio-stream = { version = "0.1.19" }
tokio-test = "0.4.6"
tokio-util = { version = "0.7.20" }
tonic = { version = "0.14.9" }
tonic-prost = { version = "0.14.9" }
tonic-prost-build = { version = "0.14.9" }
tonic = { version = "0.14.8" }
tonic-prost = { version = "0.14.8" }
tonic-prost-build = { version = "0.14.8" }
tower = { version = "0.5.3" }
tower-http = { version = "0.7.1" }
h3 = "0.0.8"
@@ -319,13 +326,17 @@ rustfs-erasure-codec = { version = "9.0.0" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.34.0" }
redis = { version = "1.8.1" }
redis = { version = "1.8.0" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.6" }
rust-embed = { version = "8.13.0" }
rustc-hash = { version = "2.1.3" }
s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
# The same s3s pin under the name the legacy edge of the gateway migration uses
# (crates/s3-types, feature `compat-s3s`). Keep `rev` identical to `s3s` above.
# DELETE BY T4.2 (rustfs/backlog#2784).
legacy-s3s = { package = "s3s", git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.4"
@@ -334,7 +345,8 @@ compact_str = "0.10.0"
snap = "1.1.2"
starshard = { version = "2.4.0" }
strum = { version = "0.28.0" }
sysinfo = "0.40.1"
sync_wrapper = "1.0.2"
sysinfo = "0.40.0"
temp-env = "0.3.6"
tempfile = "3.27.0"
test-case = "3.4.0"
@@ -386,7 +398,7 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
rustfs-mimalloc = { version = "0.7.2" }
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
hotpath = { version = "0.30.2", default-features = false }
hotpath = { version = "0.30.1", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.49" }
+94 -1
View File
@@ -17,7 +17,8 @@
//! when PublicAccessBlock configuration is missing or explicitly set.
use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
use aws_sdk_s3::types::PublicAccessBlockConfiguration;
use crate::sts_query_compat_test::{OPA_AUTH_TOKEN, OpaMock};
use aws_sdk_s3::types::{PublicAccessBlockConfiguration, Tag, Tagging};
use tracing::info;
async fn setup_public_bucket(
@@ -69,6 +70,98 @@ async fn anonymous_get_object(
local_http_client().get(&url).send().await
}
#[tokio::test]
async fn test_plain_http_matches_secure_transport_false_bucket_policy() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server(vec![]).await?;
assert!(env.url.starts_with("http://"), "fixture must exercise plaintext HTTP: {}", env.url);
let bucket = "secure-transport-policy";
let key = "plain-http.txt";
let admin = env.create_s3_client();
admin.create_bucket().bucket(bucket).send().await?;
admin
.put_object()
.bucket(bucket)
.key(key)
.body(aws_sdk_s3::primitives::ByteStream::from_static(b"plain http"))
.send()
.await?;
admin
.put_bucket_policy()
.bucket(bucket)
.policy(
serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": format!("arn:aws:s3:::{bucket}/*"),
"Condition": {"Bool": {"aws:SecureTransport": "false"}}
}]
})
.to_string(),
)
.send()
.await?;
let response = anonymous_get_object(&env, bucket, key).await?;
assert_eq!(response.status().as_u16(), 200);
assert_eq!(response.bytes().await?.as_ref(), b"plain http");
Ok(())
}
/// Root credentials have no attached IAM policy, so these requests depend on
/// the owner flag reaching policy evaluation unchanged.
#[tokio::test]
async fn test_root_requests_keep_policy_owner_flag() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
let opa = OpaMock::start().await?;
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_with_env(
vec![],
&[
("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
],
)
.await?;
let bucket = "root-owner-flag";
let key = "owner.txt";
let client = env.create_s3_client();
client.create_bucket().bucket(bucket).send().await?;
client
.put_object()
.bucket(bucket)
.key(key)
.body(aws_sdk_s3::primitives::ByteStream::from_static(b"owner access"))
.send()
.await?;
// Duplicate stored tags are ambiguous. An owner request must not enter the
// OPA tag-loading path, which rejects this metadata before evaluation.
client
.put_bucket_tagging()
.bucket(bucket)
.tagging(
Tagging::builder()
.tag_set(Tag::builder().key("department").value("finance").build()?)
.tag_set(Tag::builder().key("department").value("engineering").build()?)
.build()?,
)
.send()
.await?;
let object = client.get_object().bucket(bucket).key(key).send().await?;
assert_eq!(object.body.collect().await?.into_bytes().as_ref(), b"owner access");
Ok(())
}
/// Issue #2036: Anonymous GetObject should succeed when bucket policy allows it
/// and no PublicAccessBlock configuration exists (ConfigNotFound).
#[tokio::test]
@@ -16,193 +16,8 @@
mod tests {
use crate::common::{RustFSTestEnvironment, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::config::http::HttpResponse;
use aws_sdk_s3::error::{ProvideErrorMetadata, SdkError};
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, VersioningConfiguration};
fn assert_not_found<E: ProvideErrorMetadata + std::fmt::Debug>(error: &SdkError<E, HttpResponse>) {
assert_eq!(error.raw_response().map(|r| r.status().as_u16()), Some(404), "{error:?}");
assert_eq!(
error.as_service_error().and_then(ProvideErrorMetadata::code),
Some("NoSuchKey"),
"{error:?}"
);
}
#[tokio::test]
async fn conditional_writes_cannot_replace_delete_marker_with_if_match() {
init_logging();
let mut env = RustFSTestEnvironment::new().await.expect("create test environment");
env.start_rustfs_server(vec![]).await.expect("start RustFS");
let client = env.create_s3_client();
let bucket = "conditional-delete-marker-proof";
create_versioned_bucket(&client, bucket).await;
client
.put_object()
.bucket(bucket)
.key("source")
.body(ByteStream::from_static(b"source"))
.send()
.await
.expect("write copy source");
for operation in ["put", "copy", "multipart"] {
let old = client
.put_object()
.bucket(bucket)
.key(operation)
.body(ByteStream::from_static(b"old"))
.send()
.await
.expect("write historical version");
let old_etag = old.e_tag().expect("historical ETag");
let marker = client
.delete_object()
.bucket(bucket)
.key(operation)
.send()
.await
.expect("create delete marker");
let marker_id = marker.version_id().expect("delete marker version");
let (upload_id, completed) = if operation == "multipart" {
let upload = client
.create_multipart_upload()
.bucket(bucket)
.key(operation)
.send()
.await
.expect("initiate multipart upload");
let upload_id = upload.upload_id().expect("upload ID").to_owned();
let part = client
.upload_part()
.bucket(bucket)
.key(operation)
.upload_id(&upload_id)
.part_number(1)
.body(ByteStream::from_static(b"new"))
.send()
.await
.expect("upload part");
let completed = CompletedMultipartUpload::builder()
.parts(
CompletedPart::builder()
.part_number(1)
.set_e_tag(part.e_tag().map(str::to_owned))
.build(),
)
.build();
(Some(upload_id), Some(completed))
} else {
(None, None)
};
for etag in [old_etag, "\"00000000000000000000000000000000\""] {
match operation {
"put" => {
let err = client
.put_object()
.bucket(bucket)
.key(operation)
.if_match(etag)
.body(ByteStream::from_static(b"new"))
.send()
.await
.expect_err("If-Match must reject marker");
assert_not_found(&err);
}
"copy" => {
let err = client
.copy_object()
.bucket(bucket)
.key(operation)
.copy_source(format!("{bucket}/source"))
.if_match(etag)
.send()
.await
.expect_err("If-Match copy must reject marker");
assert_not_found(&err);
}
_ => {
let err = client
.complete_multipart_upload()
.bucket(bucket)
.key(operation)
.upload_id(upload_id.as_deref().expect("upload ID"))
.multipart_upload(completed.clone().expect("completed parts"))
.if_match(etag)
.send()
.await
.expect_err("If-Match completion must reject marker");
assert_not_found(&err);
}
}
let versions = client
.list_object_versions()
.bucket(bucket)
.prefix(operation)
.send()
.await
.expect("list versions after rejected write");
assert_eq!(versions.versions().len(), 1, "{operation}: failed write created a version");
assert_eq!(versions.delete_markers().len(), 1);
assert_eq!(versions.delete_markers()[0].version_id(), Some(marker_id));
assert_eq!(versions.delete_markers()[0].is_latest(), Some(true));
}
match operation {
"put" => {
client
.put_object()
.bucket(bucket)
.key(operation)
.if_none_match("*")
.body(ByteStream::from_static(b"new"))
.send()
.await
.expect("If-None-Match PUT over marker");
}
"copy" => {
client
.copy_object()
.bucket(bucket)
.key(operation)
.copy_source(format!("{bucket}/source"))
.if_none_match("*")
.send()
.await
.expect("If-None-Match copy over marker");
}
_ => {
client
.complete_multipart_upload()
.bucket(bucket)
.key(operation)
.upload_id(upload_id.as_deref().expect("upload ID"))
.multipart_upload(completed.expect("completed parts"))
.if_none_match("*")
.send()
.await
.expect("If-None-Match completion over marker");
}
}
let current = client
.get_object()
.bucket(bucket)
.key(operation)
.send()
.await
.expect("read successful conditional write")
.body
.collect()
.await
.expect("read object body")
.into_bytes();
let expected: &[u8] = if operation == "copy" { b"source" } else { b"new" };
assert_eq!(current.as_ref(), expected, "{operation}: unexpected current object");
}
}
use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};
async fn create_versioned_bucket(client: &Client, bucket: &str) {
client
@@ -1636,53 +1636,20 @@ async fn hot_window_with_old_followers(
let mixed_initial_physical = if mixed { json!(converged_shards(&run)?) } else { Value::Null };
let scheduler_config = run.assert_default_cadence().await?;
let baseline = run.quiet_baseline().await?;
// A leader-lock refresh gap during old-binary replacement can make
// `leader()` briefly observe a different node. Poll until the coordinator
// is stable before comparing against the measured fleet. A persistent
// change to an old-binary node is a genuine election outcome and fails.
let (leader, leader_pid) = if mixed {
let expected_leader = usize::try_from(
mixed_fleet["leader"]
.as_u64()
.ok_or("mixed fleet omitted its measured leader")?,
)?;
let expected_pid = mixed_fleet["leader_pid"]
.as_u64()
.ok_or("mixed fleet omitted its measured leader pid")?;
let mut resolved = None;
for _ in 0..20 {
let candidate = run.leader().await?;
let candidate_pid = u64::from(run.pid(candidate)?);
if Some(u64::try_from(candidate)?) == mixed_fleet["leader"].as_u64() && Some(candidate_pid) == Some(expected_pid) {
resolved = Some((candidate, run.pid(candidate)?));
break;
}
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
}
match resolved {
Some(pair) => pair,
None => {
let current = match run.leader().await {
Ok(node) => Some((node, run.pid(node).ok())),
Err(_) => None,
};
return Err(format!(
"the hot window did not retain its measured new coordinator after stabilization \
(expected node {expected_leader} pid {expected_pid}, observed {current:?})"
)
.into());
}
}
} else {
let leader = run.leader().await?;
(leader, run.pid(leader)?)
};
let leader = run.leader().await?;
let leader_pid = run.pid(leader)?;
let producer = (leader + 1) % 4;
let producers = if fault.is_some() {
vec![producer, (leader + 2) % 4]
} else {
vec![producer]
};
if mixed
&& (mixed_fleet["leader"].as_u64() != Some(u64::try_from(leader)?)
|| mixed_fleet["leader_pid"].as_u64() != Some(u64::from(leader_pid)))
{
return Err("the hot window did not retain its measured new coordinator".into());
}
let producer_pid = run.pid(producer)?;
let producer_address = run.peer_address(producer);
let [cold_a, cold_b, hot] = run.buckets.clone();
@@ -0,0 +1,512 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! What one stack answered, and how two answers are compared.
//!
//! An [`Observation`] keeps only what a client acts on: the HTTP status, the S3 error code, the
//! allowlisted response headers and a few typed fields decoded from the response. The error
//! message is kept for diagnostics and never compared: message wording is judged by the wire
//! corpus, not here. The comparator is pure so it can be tested without a server.
use serde::Serialize;
use std::collections::{BTreeMap, BTreeSet};
use std::path::{Path, PathBuf};
/// Response headers compared exactly (after [`normalized_header_value`]).
const HEADER_ALLOWLIST: [&str; 7] = [
"content-type",
"x-amz-version-id",
"etag",
"x-amz-delete-marker",
"content-range",
"accept-ranges",
"x-amz-request-charged",
];
/// Every response header with this prefix is compared as well.
const HEADER_ALLOWLIST_PREFIX: &str = "x-amz-server-side-encryption";
/// Placeholder for a field one side did not send.
const ABSENT: &str = "<absent>";
/// The lowercase header name if the comparator looks at this header.
pub(crate) fn compared_header(name: &str) -> Option<String> {
let name = name.to_ascii_lowercase();
(HEADER_ALLOWLIST.contains(&name.as_str()) || name.starts_with(HEADER_ALLOWLIST_PREFIX)).then_some(name)
}
/// A version ID is minted per process, so its value never matches across two servers; only
/// whether it is the literal `null` (an unversioned write) carries meaning.
fn normalized_header_value(name: &str, value: &str) -> String {
if name == "x-amz-version-id" && value != "null" {
"<version-id>".to_owned()
} else {
value.to_owned()
}
}
/// One request's outcome as a client sees it.
#[derive(Clone, Debug, Serialize)]
pub(crate) struct Observation {
/// The S3 operation, named as in the `s3s::S3` trait (`GetObject`).
pub op: &'static str,
/// Which request of the script this was (`normal`, `negative`, or a setup label).
pub case: &'static str,
/// HTTP status; 0 when no response arrived.
pub status: u16,
/// The S3 error code, or the SDK's error kind when there was no service error.
pub error_code: Option<String>,
/// Allowlisted headers only, lowercase names, normalized values.
pub headers: BTreeMap<String, String>,
/// Typed values decoded from the response body, never free text.
pub fields: BTreeMap<String, String>,
/// Diagnostic only; never compared.
#[serde(skip)]
pub message: Option<String>,
}
impl Observation {
/// Builds an observation, keeping only the allowlisted headers.
pub(crate) fn new<'h>(
op: &'static str,
case: &'static str,
status: u16,
error_code: Option<String>,
headers: impl IntoIterator<Item = (&'h str, &'h str)>,
) -> Self {
let headers = headers
.into_iter()
.filter_map(|(name, value)| compared_header(name).map(|name| (name, value)))
.map(|(name, value)| {
let value = normalized_header_value(&name, value);
(name, value)
})
.collect();
Self {
op,
case,
status,
error_code,
headers,
fields: BTreeMap::new(),
message: None,
}
}
pub(crate) fn with_field(mut self, name: &str, value: impl Into<String>) -> Self {
self.fields.insert(name.to_owned(), value.into());
self
}
pub(crate) fn is_success(&self) -> bool {
(200..300).contains(&self.status)
}
}
/// One field on which the two stacks answered differently.
#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
pub(crate) struct Difference {
pub op: String,
pub case: String,
pub field: String,
pub a: String,
pub b: String,
}
/// Every difference between two transcripts of the same script, matched by `(op, case)`.
pub(crate) fn compare(a: &[Observation], b: &[Observation]) -> Vec<Difference> {
let mut differences = Vec::new();
let (index_a, duplicates_a) = index(a);
let (index_b, duplicates_b) = index(b);
for (op, case) in duplicates_a.union(&duplicates_b) {
let count = |side: &[Observation]| {
side.iter()
.filter(|observation| (observation.op, observation.case) == (*op, *case))
.count()
.to_string()
};
differences.push(difference(op, case, "duplicate_case", count(a), count(b)));
}
let keys: BTreeSet<_> = index_a.keys().chain(index_b.keys()).copied().collect();
for key @ (op, case) in keys {
match (index_a.get(&key), index_b.get(&key)) {
(Some(a), Some(b)) => compare_one(a, b, &mut differences),
(a, b) => {
let presence = |side: Option<&&Observation>| if side.is_some() { "present" } else { ABSENT }.to_owned();
differences.push(difference(op, case, "observation", presence(a), presence(b)));
}
}
}
differences
}
type Key = (&'static str, &'static str);
fn index(side: &[Observation]) -> (BTreeMap<Key, &Observation>, BTreeSet<Key>) {
let mut index = BTreeMap::new();
let mut duplicates = BTreeSet::new();
for observation in side {
let key = (observation.op, observation.case);
if index.insert(key, observation).is_some() {
duplicates.insert(key);
}
}
(index, duplicates)
}
fn difference(op: &str, case: &str, field: &str, a: String, b: String) -> Difference {
Difference {
op: op.to_owned(),
case: case.to_owned(),
field: field.to_owned(),
a,
b,
}
}
fn compare_one(a: &Observation, b: &Observation, differences: &mut Vec<Difference>) {
let mut field = |name: String, value_a: Option<&str>, value_b: Option<&str>| {
if value_a != value_b {
let show = |value: Option<&str>| value.unwrap_or(ABSENT).to_owned();
differences.push(difference(a.op, a.case, &name, show(value_a), show(value_b)));
}
};
field(
"status".to_owned(),
Some(a.status.to_string().as_str()),
Some(b.status.to_string().as_str()),
);
field("error_code".to_owned(), a.error_code.as_deref(), b.error_code.as_deref());
let headers: BTreeSet<_> = a.headers.keys().chain(b.headers.keys()).collect();
for name in headers {
field(
format!("header.{name}"),
a.headers.get(name).map(String::as_str),
b.headers.get(name).map(String::as_str),
);
}
let fields: BTreeSet<_> = a.fields.keys().chain(b.fields.keys()).collect();
for name in fields {
field(
format!("body.{name}"),
a.fields.get(name).map(String::as_str),
b.fields.get(name).map(String::as_str),
);
}
}
/// Operations observed on both sides.
pub(crate) fn compared_ops(a: &[Observation], b: &[Observation]) -> BTreeSet<&'static str> {
let ops_b: BTreeSet<_> = b.iter().map(|observation| observation.op).collect();
a.iter()
.map(|observation| observation.op)
.filter(|op| ops_b.contains(op))
.collect()
}
/// `(op, case)` pairs observed on both sides.
pub(crate) fn compared_cases(a: &[Observation], b: &[Observation]) -> usize {
let cases_b: BTreeSet<_> = b.iter().map(|observation| (observation.op, observation.case)).collect();
a.iter()
.map(|observation| (observation.op, observation.case))
.collect::<BTreeSet<_>>()
.intersection(&cases_b)
.count()
}
/// The JSON report written for every run.
#[derive(Debug, Serialize)]
pub(crate) struct Report {
pub schema: u32,
pub run: String,
pub stack_a: String,
pub stack_b: String,
pub region_a: Option<String>,
pub region_b: Option<String>,
pub compared_ops: usize,
pub compared_cases: usize,
pub differences: Vec<Difference>,
}
impl Report {
/// Writes the report as `<directory>/<run>.json` and returns its path.
pub(crate) fn write_to(&self, directory: &Path) -> std::io::Result<PathBuf> {
std::fs::create_dir_all(directory)?;
let path = directory.join(format!("{}.json", self.run));
let json = serde_json::to_vec_pretty(self).map_err(std::io::Error::other)?;
std::fs::write(&path, json)?;
Ok(path)
}
}
/// One line per difference, for a failing assertion.
pub(crate) fn render(differences: &[Difference]) -> String {
differences
.iter()
.map(|d| format!("({}, {}, {}, a={:?}, b={:?})", d.op, d.case, d.field, d.a, d.b))
.collect::<Vec<_>>()
.join("\n")
}
#[cfg(test)]
mod tests {
use super::*;
fn ok(op: &'static str) -> Observation {
Observation::new(op, "normal", 200, None, [("content-type", "application/xml")])
}
fn fields(differences: &[Difference]) -> Vec<&str> {
differences.iter().map(|d| d.field.as_str()).collect()
}
#[test]
fn identical_transcripts_have_no_difference() {
let a = vec![
ok("HeadBucket"),
ok("GetBucketLocation").with_field("location_constraint", "us-east-1"),
];
let b = a.clone();
assert_eq!(compare(&a, &b), Vec::new());
assert_eq!(compared_ops(&a, &b).len(), 2);
}
#[test]
fn a_status_difference_is_reported() {
let a = vec![ok("PutBucketAcl")];
let b = vec![Observation::new(
"PutBucketAcl",
"normal",
403,
None,
[("content-type", "application/xml")],
)];
assert_eq!(
compare(&a, &b),
vec![Difference {
op: "PutBucketAcl".into(),
case: "normal".into(),
field: "status".into(),
a: "200".into(),
b: "403".into(),
}]
);
}
#[test]
fn an_error_code_difference_is_reported() {
let a = vec![Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), [])];
let b = vec![Observation::new(
"GetObject",
"negative",
404,
Some("NoSuchBucket".into()),
[],
)];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["error_code"]);
assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("NoSuchKey", "NoSuchBucket"));
}
#[test]
fn an_error_code_on_one_side_only_is_reported() {
let a = vec![Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), [])];
let b = vec![Observation::new("HeadBucket", "negative", 404, None, [])];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["error_code"]);
assert_eq!(differences[0].b, ABSENT);
}
#[test]
fn an_allowlisted_header_difference_is_reported() {
let a = vec![Observation::new("HeadObject", "normal", 200, None, [("ETag", "\"aa\"")])];
let b = vec![Observation::new("HeadObject", "normal", 200, None, [("etag", "\"bb\"")])];
assert_eq!(fields(&compare(&a, &b)), ["header.etag"]);
}
#[test]
fn a_header_sent_by_one_side_only_is_reported() {
let a = vec![Observation::new(
"DeleteObject",
"normal",
204,
None,
[("x-amz-delete-marker", "true")],
)];
let b = vec![Observation::new("DeleteObject", "normal", 204, None, [])];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["header.x-amz-delete-marker"]);
assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("true", ABSENT));
}
#[test]
fn every_server_side_encryption_header_is_compared() {
let header = "x-amz-server-side-encryption-customer-key-md5";
let a = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "one")])];
let b = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "two")])];
assert_eq!(fields(&compare(&a, &b)), [format!("header.{header}")]);
}
#[test]
fn headers_outside_the_allowlist_are_not_compared() {
let a = vec![Observation::new(
"ListBuckets",
"normal",
200,
None,
[
("date", "Mon"),
("x-amz-request-id", "1"),
("server", "a"),
("content-length", "10"),
],
)];
let b = vec![Observation::new(
"ListBuckets",
"normal",
200,
None,
[
("date", "Tue"),
("x-amz-request-id", "2"),
("server", "b"),
("content-length", "11"),
],
)];
assert_eq!(compare(&a, &b), Vec::new());
assert!(a[0].headers.is_empty(), "non-allowlisted headers were kept: {:?}", a[0].headers);
}
#[test]
fn distinct_version_ids_are_not_a_difference() {
let a = vec![Observation::new(
"PutObject",
"normal",
200,
None,
[("x-amz-version-id", "0b1c")],
)];
let b = vec![Observation::new(
"PutObject",
"normal",
200,
None,
[("x-amz-version-id", "9f8e")],
)];
assert_eq!(compare(&a, &b), Vec::new());
}
#[test]
fn a_null_version_id_against_a_real_one_is_reported() {
let a = vec![Observation::new(
"PutObject",
"normal",
200,
None,
[("x-amz-version-id", "null")],
)];
let b = vec![Observation::new(
"PutObject",
"normal",
200,
None,
[("x-amz-version-id", "9f8e")],
)];
assert_eq!(fields(&compare(&a, &b)), ["header.x-amz-version-id"]);
}
#[test]
fn a_body_field_difference_is_reported() {
let a = vec![ok("GetBucketLocation").with_field("location_constraint", "us-east-1")];
let b = vec![ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["body.location_constraint"]);
assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("us-east-1", "eu-west-1"));
}
#[test]
fn a_body_field_on_one_side_only_is_reported() {
let a = vec![ok("ListObjectsV2").with_field("keys", "[]")];
let b = vec![ok("ListObjectsV2")];
assert_eq!(fields(&compare(&a, &b)), ["body.keys"]);
}
#[test]
fn error_messages_are_never_compared() {
let mut a = Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), []);
let mut b = a.clone();
a.message = Some("The specified key does not exist.".into());
b.message = Some("Object not found".into());
assert_eq!(compare(&[a], &[b]), Vec::new());
}
#[test]
fn a_case_observed_on_one_side_only_is_reported() {
let a = vec![ok("HeadBucket"), ok("ListBuckets")];
let b = vec![ok("HeadBucket")];
let differences = compare(&a, &b);
assert_eq!(fields(&differences), ["observation"]);
assert_eq!(differences[0].op, "ListBuckets");
assert_eq!(compared_ops(&a, &b).into_iter().collect::<Vec<_>>(), ["HeadBucket"]);
}
#[test]
fn a_case_recorded_twice_is_reported() {
let a = vec![ok("HeadBucket"), ok("HeadBucket")];
let b = vec![ok("HeadBucket")];
assert_eq!(fields(&compare(&a, &b)), ["duplicate_case"]);
}
#[test]
fn compared_ops_counts_operations_not_cases() {
let negative = Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), []);
let a = vec![ok("HeadBucket"), negative, ok("ListBuckets")];
let b = a.clone();
assert_eq!(compared_ops(&a, &b).len(), 2);
}
#[test]
fn the_report_lists_each_difference_as_op_field_a_b() {
let directory = std::env::temp_dir().join(format!("dual-stack-report-{}", uuid::Uuid::new_v4()));
let report = Report {
schema: 1,
run: "unit".into(),
stack_a: "legacy".into(),
stack_b: "legacy".into(),
region_a: None,
region_b: Some("eu-west-1".into()),
compared_ops: 1,
compared_cases: 1,
differences: compare(
&[ok("GetBucketLocation").with_field("location_constraint", "us-east-1")],
&[ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")],
),
};
let path = report.write_to(&directory).expect("write the report");
let json: serde_json::Value = serde_json::from_slice(&std::fs::read(&path).expect("read the report")).expect("parse");
let _ = std::fs::remove_dir_all(&directory);
assert_eq!(path.file_name().and_then(|name| name.to_str()), Some("unit.json"));
assert_eq!(json["compared_ops"], 1);
assert_eq!(
json["differences"],
serde_json::json!([{
"op": "GetBucketLocation",
"case": "normal",
"field": "body.location_constraint",
"a": "us-east-1",
"b": "eu-west-1",
}])
);
}
}
@@ -0,0 +1,199 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Dual-stack differential: one `rustfs` binary, two processes, one SDK script (rustfs/backlog#2740).
//!
//! Stack A always runs `RUSTFS_S3_STACK=legacy`. Stack B runs the value of `RUSTFS_E2E_STACK_B`
//! (default `legacy` until the gateway stack is assembled), and `RUSTFS_E2E_STACK_B_REGION`, when
//! set, starts B in another region so that a difference can be injected by hand. Both servers
//! answer the script in `scenarios`, which covers every operation of `impl S3 for FS`; `compare`
//! reports each `(op, field, a, b)` on which the answers differ, and every run writes its report
//! to `<target>/dual-stack/<run>.json`. Two processes, not two embedded servers: bucket metadata
//! is process-global, so two stacks in one process would read each other's buckets.
mod compare;
mod scenarios;
use crate::common::{RustFSTestEnvironment, init_logging, workspace_root};
use compare::{Difference, Observation, Report, compare, compared_cases, compared_ops, render};
use std::path::PathBuf;
use std::time::{SystemTime, UNIX_EPOCH};
type BoxError = Box<dyn std::error::Error + Send + Sync>;
const STACK_A: &str = "legacy";
const ENV_STACK_B: &str = "RUSTFS_E2E_STACK_B";
const ENV_STACK_B_REGION: &str = "RUSTFS_E2E_STACK_B_REGION";
/// Becomes `gateway` once rustfs/backlog#2734 T2.5 assembles the gateway stack.
const DEFAULT_STACK_B: &str = "legacy";
/// The region the negative control gives stack A explicitly, so an inherited `RUSTFS_REGION`
/// cannot erase the injected difference. Not `us-east-1`: that region answers GetBucketLocation
/// with an empty LocationConstraint and re-creating an owned bucket with 200, so it would not
/// report its own name and would add CreateBucket differences to the control.
const CONTROL_REGION_A: &str = "us-west-2";
const CONTROL_REGION_B: &str = "eu-west-1";
/// `normal` cases the legacy stack must answer with a 2xx. The comparison alone cannot tell a
/// script that exercises real behavior from one in which both stacks fail the same way. Five of
/// the 73 fail on legacy by design of this script: PutBucketReplication (no registered remote
/// target), GetBucketReplication (so no configuration), PutBucketNotificationConfiguration (no
/// notification target configured), RestoreObject (the object is not archived) and
/// GetObjectTorrent (legacy answers `NoSuchKey`).
const MIN_NORMAL_SUCCESSES: usize = 68;
struct PairConfig {
stack_b: String,
region_a: Option<String>,
region_b: Option<String>,
}
impl PairConfig {
fn from_env() -> Self {
Self {
stack_b: std::env::var(ENV_STACK_B).unwrap_or_else(|_| DEFAULT_STACK_B.to_owned()),
region_a: None,
region_b: std::env::var(ENV_STACK_B_REGION).ok(),
}
}
}
/// Starts one server on `stack`, overriding its region only when asked.
async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
let mut env = RustFSTestEnvironment::new().await?;
let mut vars = vec![("RUSTFS_S3_STACK", stack)];
if let Some(region) = region {
vars.push(("RUSTFS_REGION", region));
}
env.start_rustfs_server_without_cleanup_with_env(&vars).await?;
Ok(env)
}
/// Starts stack A and stack B side by side, each with its own port and data directory.
async fn spawn_pair(config: &PairConfig) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment), BoxError> {
tokio::try_join!(
start_stack(STACK_A, config.region_a.as_deref()),
start_stack(&config.stack_b, config.region_b.as_deref())
)
}
fn report_directory() -> PathBuf {
let target = std::env::var_os("CARGO_TARGET_DIR").map_or_else(|| PathBuf::from("target"), PathBuf::from);
let target = if target.is_absolute() {
target
} else {
workspace_root().join(target)
};
target.join("dual-stack")
}
struct Run {
report: Report,
path: PathBuf,
a: Vec<Observation>,
}
/// Runs the script against both stacks concurrently and writes the report.
async fn run_differential(label: &str, config: &PairConfig) -> Run {
let (server_a, server_b) = spawn_pair(config).await.expect("start stack A and stack B");
let (a, b) = tokio::join!(Box::pin(scenarios::run(&server_a)), Box::pin(scenarios::run(&server_b)));
let millis = SystemTime::now()
.duration_since(UNIX_EPOCH)
.map_or(0, |elapsed| elapsed.as_millis());
let report = Report {
schema: 1,
run: format!("{label}-{millis}-{}", std::process::id()),
stack_a: STACK_A.to_owned(),
stack_b: config.stack_b.clone(),
region_a: config.region_a.clone(),
region_b: config.region_b.clone(),
compared_ops: compared_ops(&a, &b).len(),
compared_cases: compared_cases(&a, &b),
differences: compare(&a, &b),
};
let path = report.write_to(&report_directory()).expect("write the dual-stack report");
Run { report, path, a }
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn both_stacks_answer_every_operation_identically() {
init_logging();
let config = PairConfig::from_env();
let run = run_differential("self-comparison", &config).await;
let report = &run.report;
let mut observed: Vec<_> = run.a.iter().map(|observation| observation.op).collect();
observed.sort_unstable();
observed.dedup();
assert_eq!(
observed,
scenarios::OPERATIONS,
"the script must drive exactly the operations of impl S3 for FS"
);
assert_eq!(report.compared_ops, scenarios::OPERATIONS.len(), "report: {}", run.path.display());
assert!(
report.differences.is_empty(),
"stack {} and stack {} differ on {} of {} cases (report: {}):\n{}",
report.stack_a,
report.stack_b,
report.differences.len(),
report.compared_cases,
run.path.display(),
render(&report.differences)
);
let not_4xx: Vec<_> = run
.a
.iter()
.filter(|observation| observation.case == "negative" && !(400..500).contains(&observation.status))
.collect();
assert!(not_4xx.is_empty(), "negative cases must fail with a 4xx on {STACK_A}: {not_4xx:#?}");
let successes = run
.a
.iter()
.filter(|observation| observation.case == "normal" && observation.is_success())
.count();
assert!(
successes >= MIN_NORMAL_SUCCESSES,
"only {successes} normal cases succeeded on {STACK_A}: {:#?}",
run.a
);
}
#[tokio::test]
async fn differential_detects_an_injected_difference() {
init_logging();
let config = PairConfig {
stack_b: STACK_A.to_owned(),
region_a: Some(CONTROL_REGION_A.to_owned()),
region_b: Some(CONTROL_REGION_B.to_owned()),
};
let run = run_differential("injected-region", &config).await;
let expected = Difference {
op: "GetBucketLocation".to_owned(),
case: "normal".to_owned(),
field: "body.location_constraint".to_owned(),
a: CONTROL_REGION_A.to_owned(),
b: CONTROL_REGION_B.to_owned(),
};
assert!(
run.report.differences.contains(&expected),
"B started in {CONTROL_REGION_B} but GetBucketLocation was not reported (report: {}):\n{}",
run.path.display(),
render(&run.report.differences)
);
}
}
File diff suppressed because it is too large Load Diff
+6
View File
@@ -31,6 +31,12 @@ pub mod chaos;
#[cfg(test)]
pub mod fake_s3_target;
// Dual-stack differential (rustfs/backlog#2740): one binary, two processes on
// different RUSTFS_S3_STACK values, the same SDK script for every S3 operation,
// compared on status, error code, allowlisted headers and typed body fields.
#[cfg(test)]
mod dual_stack_differential;
// On-demand migration (backlog#2147): shared two-server environment, admin
// wrappers, and the harness self-test (backlog#2151). Behavior scenarios are
// added by later ODM tasks.
+78 -1
View File
@@ -32,7 +32,9 @@
//! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-3p3x-734c-h5vx>
use crate::common::rustfs_binary_path_with_features;
use crate::common::{AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via};
use crate::common::{
AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, signed_s3_request,
};
use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, admin_request};
use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
use anyhow::Result;
@@ -774,6 +776,81 @@ async fn test_webdav_core_operations_direct() -> Result<()> {
test_webdav_core_operations().await
}
#[tokio::test]
async fn test_webdav_content_type_directory_marker() -> Result<()> {
let env = ProtocolTestEnvironment::new().map_err(|e| anyhow::anyhow!("{}", e))?;
let admin_base_url = format!("http://{}", S3_TEST_ADDRESS);
let binary_path = rustfs_binary_path_with_features(Some("webdav"));
let mut server_process = Command::new(&binary_path)
.arg("--address")
.arg(S3_TEST_ADDRESS)
.env("RUSTFS_CONSOLE_ENABLE", "false")
.env("RUSTFS_WEBDAV_ENABLE", "true")
.env("RUSTFS_WEBDAV_ADDRESS", WEBDAV_ADDRESS)
.env("RUSTFS_WEBDAV_TLS_ENABLED", "false")
.arg(&env.temp_dir)
.spawn()?;
let result = async {
ProtocolTestEnvironment::wait_for_port_ready(WEBDAV_PORT, 30)
.await
.map_err(|e| anyhow::anyhow!("{}", e))?;
let client = create_client();
let bucket_name = "webdav-content-type-bucket";
let directory_name = "content-type-directory";
let webdav_base_url = format!("http://{}", WEBDAV_ADDRESS);
let mkcol_bucket = client
.request(
reqwest::Method::from_bytes(b"MKCOL").expect("MKCOL is a valid HTTP method"),
format!("{}/{}", webdav_base_url, bucket_name),
)
.header("Authorization", basic_auth_header())
.send()
.await?;
anyhow::ensure!(
mkcol_bucket.status().is_success() || mkcol_bucket.status().as_u16() == 201,
"MKCOL bucket should succeed, got: {}",
mkcol_bucket.status()
);
let mkcol_directory = client
.request(
reqwest::Method::from_bytes(b"MKCOL").expect("MKCOL is a valid HTTP method"),
format!("{}/{}/{}", webdav_base_url, bucket_name, directory_name),
)
.header("Authorization", basic_auth_header())
.send()
.await?;
anyhow::ensure!(
mkcol_directory.status().is_success() || mkcol_directory.status().as_u16() == 201,
"MKCOL directory should succeed, got: {}",
mkcol_directory.status()
);
let head_url = format!("{}/{}/{}/", admin_base_url, bucket_name, directory_name);
let head = signed_s3_request(http::Method::HEAD, &head_url, None, None, DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY)
.await
.map_err(|error| anyhow::anyhow!("{error}"))?;
anyhow::ensure!(head.status().is_success(), "HEAD directory marker should succeed, got: {}", head.status());
let content_type = head
.headers()
.get("content-type")
.and_then(|value| value.to_str().ok())
.unwrap_or_default();
anyhow::ensure!(
content_type == "application/x-directory",
"WebDAV directory marker must preserve its content type, got: {content_type}"
);
Ok(())
}
.await;
let _ = server_process.kill().await;
let _ = server_process.wait().await;
result
}
/// Extract the numeric value of the first occurrence of a DAV property in a
/// PROPFIND multistatus body. Namespace-prefix agnostic; a self-closing
/// (unreported) property yields None.
-1
View File
@@ -2146,4 +2146,3 @@ mod tests {
);
}
}
// event-drill: isolated harness-only change
@@ -1112,9 +1112,8 @@ async fn wait_for_upgrade_writes(
match response {
Ok(Ok(_)) => break,
Ok(Err(error)) => {
let code = error.as_service_error().and_then(ProvideErrorMetadata::code);
if error.raw_response().map(|response| response.status().as_u16()) != Some(503)
|| !matches!(code, Some("ServiceUnavailable" | "SlowDown" | "SlowDownRead"))
|| error.as_service_error().and_then(ProvideErrorMetadata::code) != Some("ServiceUnavailable")
{
return Err(format!("{phase}: node {node} {probe} failed: {error:?}").into());
}
+2 -1
View File
@@ -139,6 +139,7 @@ rustfs-lifecycle.workspace = true
rustfs-s3-client = { workspace = true }
rustfs-s3-types = { workspace = true }
rustfs-data-usage.workspace = true
rustfs-gateway-types.workspace = true
rustfs-object-capacity.workspace = true
rustfs-object-data-cache = { workspace = true, features = ["runtime-memory"] }
arc-swap.workspace = true
@@ -157,7 +158,6 @@ serde = { workspace = true, features = ["derive"] }
time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
bytesize.workspace = true
serde_json = { workspace = true, features = ["raw_value"] }
s3s = { workspace = true, features = ["minio"] }
http.workspace = true
opentelemetry.workspace = true
http-body = { workspace = true }
@@ -248,6 +248,7 @@ windows-sys = { workspace = true, features = [
windows-sys = { workspace = true, features = ["Win32_System_Ioctl"] }
[dev-dependencies]
s3s = { workspace = true, features = ["minio"] }
aws-smithy-async.workspace = true
rustfs-s3-ops.workspace = true
tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] }
+8 -6
View File
@@ -66,6 +66,8 @@ pub mod bucket {
}
pub mod bucket_lifecycle_ops {
pub use rustfs_lifecycle::{RestoreOutputLocation, RestoreRequest, RestoreS3Location, RestoreTag, RestoreTagging};
pub use crate::bucket::lifecycle::bucket_lifecycle_ops::{
ExpiryState, LifecycleOps, ManualTransitionCancelCheck, ManualTransitionProgressSink,
ManualTransitionQueueSnapshot, ManualTransitionRunExecution, ManualTransitionRunOptions,
@@ -137,7 +139,7 @@ pub mod bucket {
pub use crate::bucket::lifecycle::lifecycle::{
Event, ExpirationOptions, IlmAction, LIFECYCLE_MALFORMED_XML_ERROR_KIND, Lifecycle, LifecycleCalculate,
ObjectOpts, RuleValidate, TRANSITION_COMPLETE, TRANSITION_PENDING, TransitionOptions, expected_expiry_time,
object_opts_from_object_info,
format_lifecycle_date, object_opts_from_object_info, parse_lifecycle_date,
};
}
@@ -235,7 +237,7 @@ pub mod bucket {
}
pub mod object_lock {
pub use crate::bucket::object_lock::{ObjectLockApi, ObjectLockStatusExt};
pub use crate::bucket::object_lock::ObjectLockApi;
pub mod types {
pub use crate::bucket::object_lock::types::{
@@ -330,9 +332,9 @@ pub mod bucket {
pub mod utils {
pub use crate::bucket::utils::{
check_bucket_and_object_names, check_list_objs_args, check_object_name_for_length_and_slash,
check_valid_bucket_name_strict, deserialize, has_bad_path_component, is_meta_bucketname, is_valid_object_prefix,
serialize,
BucketConfigXml, BucketConfigXmlError, check_bucket_and_object_names, check_list_objs_args,
check_object_name_for_length_and_slash, check_valid_bucket_name_strict, deserialize, has_bad_path_component,
is_meta_bucketname, is_valid_object_prefix, serialize,
};
}
@@ -359,7 +361,7 @@ pub mod capacity {
}
pub mod object_api_utils {
pub use crate::object_api::object_api_utils::{ObjReaderFn, PutObjReader, get_raw_etag, new_getobjectreader, to_s3s_etag};
pub use crate::object_api::object_api_utils::{ObjReaderFn, PutObjReader, get_raw_etag, new_getobjectreader};
}
pub mod cluster {
@@ -88,12 +88,14 @@ use rustfs_filemeta::{
FileInfo, FileInfoOpts, NULL_VERSION_ID, RestoreStatus, RestoreStatusOps, TRANSITION_COMPLETE, get_file_info,
is_restored_object_on_disk,
};
use rustfs_gateway_types::persistence::PersistedObjectLockConfiguration as ObjectLockConfiguration;
use rustfs_gateway_types::persistence::{PersistedLifecycleConfiguration, PersistedLifecycleFilter};
use rustfs_lifecycle::{RestoreRequest, RestoreS3Location};
use rustfs_lock::{NamespaceLockGuard, error::LockError};
use rustfs_scanner_metrics::metrics::{
IlmAction, Metrics, ScannerLifecycleExpiryStateUpdate, ScannerLifecycleTransitionStateUpdate, global_metrics,
};
use rustfs_utils::{get_env_i64, get_env_usize, path::encode_dir_object, string::parse_bool};
use s3s::dto::{BucketLifecycleConfiguration, ExpirationStatus, ObjectLockConfiguration, RestoreRequest, RestoreRequestType};
use sha2::{Digest, Sha256};
use std::any::Any;
use std::collections::{BTreeMap, HashMap, HashSet};
@@ -300,7 +302,7 @@ impl LifecycleSys {
Arc::new(Self)
}
pub async fn get(&self, bucket: &str) -> Option<BucketLifecycleConfiguration> {
pub async fn get(&self, bucket: &str) -> Option<PersistedLifecycleConfiguration> {
match metadata_boundary::get_lifecycle_config(bucket).await {
Ok((lc, _)) => Some(lc),
Err(Error::ConfigNotFound) => None,
@@ -3557,7 +3559,7 @@ async fn stale_upload_lifecycle_due(
Err(_) => return None,
};
let has_size_filter = |filter: &s3s::dto::LifecycleRuleFilter| {
let has_size_filter = |filter: &PersistedLifecycleFilter| {
filter.object_size_greater_than.is_some()
|| filter.object_size_less_than.is_some()
|| filter
@@ -3566,7 +3568,7 @@ async fn stale_upload_lifecycle_due(
.is_some_and(|and| and.object_size_greater_than.is_some() || and.object_size_less_than.is_some())
};
let needs_size = lifecycle.rules.iter().any(|rule| {
rule.status.as_str() != ExpirationStatus::DISABLED
rule.status.as_str() != "Disabled"
&& rule.abort_incomplete_multipart_upload.is_some()
&& rule.filter.as_ref().is_some_and(has_size_filter)
});
@@ -4000,7 +4002,7 @@ pub fn init_background_stale_multipart_upload_cleanup(api: Arc<ECStore>) {
});
}
pub async fn validate_transition_tier(lc: &BucketLifecycleConfiguration) -> Result<(), std::io::Error> {
pub async fn validate_transition_tier(lc: &PersistedLifecycleConfiguration) -> Result<(), std::io::Error> {
for rule in &lc.rules {
if let Some(transitions) = &rule.transitions {
for transition in transitions {
@@ -4117,8 +4119,9 @@ pub(crate) async fn enqueue_immediate_expiry(api: Arc<ECStore>, oi: &ObjectInfo,
.map(lifecycle::object_opts_from_object_info)
.collect::<Vec<ObjectOpts>>();
let lock_config = configs.object_lock;
let evaluator_lock_config = lock_config.clone();
let events = match Evaluator::new(lifecycle)
.with_lock_retention(lock_config.clone())
.with_lock_retention(evaluator_lock_config)
.eval(&object_opts)
.await
{
@@ -4598,19 +4601,26 @@ pub async fn enqueue_transition_for_existing_objects_scoped(
}
}
fn lifecycle_rule_has_date_expiration(lc: &BucketLifecycleConfiguration, rule_id: &str) -> bool {
fn lifecycle_rule_has_date_expiration(lc: &PersistedLifecycleConfiguration, rule_id: &str) -> bool {
lc.rules.iter().any(|rule| {
rule.status == ExpirationStatus::from_static(ExpirationStatus::ENABLED)
rule.status == "Enabled"
&& rule.id.as_deref() == Some(rule_id)
&& rule.expiration.as_ref().is_some_and(|expiration| expiration.date.is_some())
})
}
fn should_defer_date_expiry_for_recent_config_update(lc: &BucketLifecycleConfiguration, now: OffsetDateTime) -> bool {
lc.expiry_updated_at.as_ref().is_some_and(|updated_at| {
let updated_at = OffsetDateTime::from(updated_at.clone());
now.unix_timestamp().saturating_sub(updated_at.unix_timestamp()) < DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS
})
fn should_defer_date_expiry_for_recent_config_update(lc: &PersistedLifecycleConfiguration, now: OffsetDateTime) -> bool {
lc.expiry_updated_at
.as_ref()
.is_some_and(|updated_at| match rustfs_lifecycle::parse_lifecycle_date(updated_at) {
Ok(updated_at) => {
now.unix_timestamp().saturating_sub(updated_at.unix_timestamp()) < DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS
}
Err(error) => {
warn!(error = %error, "Deferring date expiry with an unreadable update timestamp");
true
}
})
}
async fn apply_existing_object_expiry(
@@ -4630,7 +4640,7 @@ async fn apply_existing_object_expiry(
struct ExistingObjectExpiryContext<'a> {
api: Arc<ECStore>,
bucket: &'a str,
lc: Arc<BucketLifecycleConfiguration>,
lc: Arc<PersistedLifecycleConfiguration>,
lock_config: Option<Arc<ObjectLockConfiguration>>,
bucket_incarnation_id: Uuid,
src: &'a LcEventSrc,
@@ -4650,8 +4660,9 @@ async fn enqueue_expiry_for_existing_object_group(
.iter()
.map(lifecycle::object_opts_from_object_info)
.collect::<Vec<ObjectOpts>>();
let evaluator_lock_config = context.lock_config.clone();
let events = match Evaluator::new(context.lc.clone())
.with_lock_retention(context.lock_config.clone())
.with_lock_retention(evaluator_lock_config)
.eval(&object_opts)
.await
{
@@ -4845,7 +4856,7 @@ fn manual_transition_version_marker(oi: &ObjectInfo) -> String {
async fn enqueue_transition_with_lifecycle_report(
api: Option<Arc<ECStore>>,
oi: &ObjectInfo,
lc: &BucketLifecycleConfiguration,
lc: &PersistedLifecycleConfiguration,
src: &LcEventSrc,
options: &ManualTransitionRunOptions,
report: &mut ManualTransitionRunReport,
@@ -4911,7 +4922,7 @@ async fn enqueue_transition_with_lifecycle_report(
false
}
async fn enqueue_transition_with_lifecycle(oi: &ObjectInfo, lc: &BucketLifecycleConfiguration, src: &LcEventSrc) -> bool {
async fn enqueue_transition_with_lifecycle(oi: &ObjectInfo, lc: &PersistedLifecycleConfiguration, src: &LcEventSrc) -> bool {
let event = lc.eval(&oi.to_lifecycle_opts()).await;
match event.action {
IlmAction::TransitionAction | IlmAction::TransitionVersionAction => {
@@ -5425,12 +5436,8 @@ pub async fn post_restore_opts(version_id: &str, bucket: &str, object: &str) ->
})
}
fn select_restore_s3_location(rreq: &RestoreRequest) -> Result<Option<&s3s::dto::S3Location>, std::io::Error> {
if rreq
.type_
.as_ref()
.is_none_or(|type_| type_.as_str() != RestoreRequestType::SELECT)
{
fn select_restore_s3_location(rreq: &RestoreRequest) -> Result<Option<&RestoreS3Location>, std::io::Error> {
if rreq.type_.as_ref().is_none_or(|type_| type_.as_str() != "SELECT") {
return Ok(None);
}
let output_location = rreq
@@ -5472,27 +5479,25 @@ pub async fn put_restore_opts(
// the OutputLocation metadata instead of a write to `OutputLocation.S3`
// (backlog#1341). RestoreObject rejects SELECT at the API boundary; this
// is the fail-closed backstop for any other caller.
if rreq
.type_
.as_ref()
.is_some_and(|type_| type_.as_str() == RestoreRequestType::SELECT)
{
if rreq.type_.as_ref().is_some_and(|type_| type_.as_str() == "SELECT") {
return Err(std::io::Error::other("SELECT restore requests are not supported"));
}
for (k, v) in oi.user_defined.iter() {
meta.insert(k.to_string(), v.clone());
}
// The fleet capability gate guarantees physical copy-back support. Keep
// the completed transition identity for orphan recovery and remote cleanup.
rustfs_utils::http::metadata_compat::remove_str(&mut meta, rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_OPERATION_ID);
rustfs_utils::http::metadata_compat::remove_str(&mut meta, rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_WORKER_LOCK);
if !oi.user_tags.is_empty() {
meta.insert(AMZ_OBJECT_TAGGING.to_string(), (*oi.user_tags).clone());
}
let restore_expiry = lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), rreq.days.unwrap_or(1));
// Finalization publishes completion and its expiry under the object lock.
// The local copy-back commit must not make unfinished work eligible for expiry.
meta.insert(
metadata_keys::RESTORE.to_string(),
RestoreStatus {
is_restore_in_progress: Some(false),
restore_expiry_date: Some(restore_expiry),
is_restore_in_progress: Some(true),
restore_expiry_date: None,
}
.to_string(),
);
@@ -5506,6 +5511,7 @@ pub async fn put_restore_opts(
// computed MD5 is not the object's public plaintext ETag.
preserve_etag: oi.etag.clone(),
shard_integrity_write_mode: Some(oi.shard_integrity_write_mode()),
restore_copyback: true,
//expires: oi.expires,
..Default::default()
})
@@ -5525,6 +5531,18 @@ impl LifecycleOps for ObjectInfo {
if self.transitioned_object.status != lifecycle::TRANSITION_COMPLETE {
return false;
}
if rustfs_utils::http::metadata_compat::get_consistent_str(
&self.user_defined,
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_COPYBACK,
) == Some(rustfs_utils::http::metadata_compat::RESTORE_COPYBACK_PROTOCOL_V1)
&& self
.user_defined
.get(metadata_keys::RESTORE)
.and_then(|value| rustfs_filemeta::parse_restore_obj_status(value).ok())
.is_some_and(|status| status.is_restore_in_progress == Some(true))
{
return false;
}
!is_restored_object_on_disk(&self.user_defined)
}
}
@@ -5536,25 +5554,25 @@ pub trait RestoreRequestOps {
impl RestoreRequestOps for RestoreRequest {
fn validate(&self, _api: Arc<ECStore>) -> Result<(), std::io::Error> {
// SELECT type requires select_parameters, and vice versa
if self.type_.as_ref().is_none_or(|t| t.as_str() != RestoreRequestType::SELECT) && self.select_parameters.is_some() {
if self.type_.as_ref().is_none_or(|t| t.as_str() != "SELECT") && self.select_parameters.is_some() {
return Err(std::io::Error::other("Select parameters can only be specified with SELECT request type"));
}
if let Some(type_) = &self.type_
&& type_.as_str() == RestoreRequestType::SELECT
&& type_.as_str() == "SELECT"
&& self.select_parameters.is_none()
{
return Err(std::io::Error::other("SELECT restore request requires select parameters to be specified"));
}
// OutputLocation is only valid for SELECT requests
if self.type_.as_ref().is_none_or(|t| t.as_str() != RestoreRequestType::SELECT) && self.output_location.is_some() {
if self.type_.as_ref().is_none_or(|t| t.as_str() != "SELECT") && self.output_location.is_some() {
return Err(std::io::Error::other("OutputLocation can only be specified with SELECT request type"));
}
select_restore_s3_location(self)?;
// Days must not be specified with SELECT requests
if let Some(type_) = &self.type_
&& type_.as_str() == RestoreRequestType::SELECT
&& type_.as_str() == "SELECT"
&& self.days.is_some_and(|d| d > 0)
{
return Err(std::io::Error::other("Days cannot be specified with SELECT restore request"));
@@ -5596,7 +5614,7 @@ pub fn lifecycle_version_delete_target(oi: &ObjectInfo) -> Option<ObjectToDelete
}
pub async fn eval_action_from_lifecycle(
lc: &BucketLifecycleConfiguration,
lc: &PersistedLifecycleConfiguration,
lock_config: Option<&ObjectLockConfiguration>,
oi: &ObjectInfo,
) -> lifecycle::Event {
@@ -6301,14 +6319,19 @@ mod tests {
use rustfs_data_usage::TierStats;
use rustfs_filemeta::metadata_keys;
use rustfs_filemeta::{FileInfo, FileMeta};
use rustfs_gateway_types::dto::MetadataEntry;
use rustfs_gateway_types::persistence::{
PersistedDefaultRetention as DefaultRetention, PersistedObjectLockConfiguration as ObjectLockConfiguration,
PersistedObjectLockRule as ObjectLockRule,
};
use rustfs_gateway_types::persistence::{
PersistedLifecycleConfiguration, PersistedLifecycleExpiration, PersistedLifecycleRule,
PersistedNoncurrentVersionExpiration, PersistedTransition,
};
use rustfs_lifecycle::{RestoreOutputLocation as OutputLocation, RestoreRequest, RestoreS3Location as S3Location};
#[cfg(feature = "test-util")]
use rustfs_s3_client::transition_api::ReaderImpl;
use rustfs_scanner_metrics::metrics::{IlmAction, global_metrics};
use s3s::dto::{
BucketLifecycleConfiguration, DefaultRetention, ExpirationStatus, LifecycleExpiration, LifecycleRule, MetadataEntry,
NoncurrentVersionExpiration, ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRetentionMode, ObjectLockRule,
OutputLocation, RestoreRequest, RestoreRequestType, S3Location, Timestamp, Transition, TransitionStorageClass,
};
use serial_test::serial;
use sha2::{Digest, Sha256};
use std::collections::HashMap;
@@ -9315,7 +9338,7 @@ mod tests {
output_location,
select_parameters: None,
tier: None,
type_: Some(RestoreRequestType::from_static(RestoreRequestType::SELECT)),
type_: Some("SELECT".to_owned()),
}
}
@@ -9632,12 +9655,12 @@ mod tests {
#[test]
fn lifecycle_rule_has_date_expiration_detects_enabled_date_rule() {
let lc = BucketLifecycleConfiguration {
let lc = PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
date: Some(Timestamp::from(OffsetDateTime::now_utc())),
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
date: Some(lifecycle::format_lifecycle_date(OffsetDateTime::now_utc()).unwrap()),
..Default::default()
}),
id: Some("rule-date".to_string()),
@@ -10154,14 +10177,15 @@ mod tests {
#[test]
fn should_defer_date_expiry_for_recent_config_update_respects_grace_window() {
let now = OffsetDateTime::now_utc();
let recent = BucketLifecycleConfiguration {
expiry_updated_at: Some(Timestamp::from(now - time::Duration::seconds(1))),
let recent = PersistedLifecycleConfiguration {
expiry_updated_at: Some(lifecycle::format_lifecycle_date(now - time::Duration::seconds(1)).unwrap()),
rules: Vec::new(),
};
let stale = BucketLifecycleConfiguration {
expiry_updated_at: Some(Timestamp::from(
now - time::Duration::seconds(DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS + 1),
)),
let stale = PersistedLifecycleConfiguration {
expiry_updated_at: Some(
lifecycle::format_lifecycle_date(now - time::Duration::seconds(DATE_EXPIRY_EXISTING_OBJECTS_GRACE_SECS + 1))
.unwrap(),
),
rules: Vec::new(),
};
@@ -10289,12 +10313,12 @@ mod tests {
assert_eq!(deleted.version_id, None);
}
fn expired_delete_marker_lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn expired_delete_marker_lifecycle() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
expired_object_delete_marker: Some(true),
..Default::default()
}),
@@ -10310,12 +10334,12 @@ mod tests {
}
}
fn latest_expiration_lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn latest_expiration_lifecycle() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
days: Some(1),
..Default::default()
}),
@@ -10331,12 +10355,12 @@ mod tests {
}
}
fn current_and_noncurrent_expiration_lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn current_and_noncurrent_expiration_lifecycle() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
days: Some(30),
..Default::default()
}),
@@ -10344,7 +10368,7 @@ mod tests {
del_marker_expiration: None,
filter: None,
id: Some("expire-current-and-noncurrent".to_string()),
noncurrent_version_expiration: Some(NoncurrentVersionExpiration {
noncurrent_version_expiration: Some(PersistedNoncurrentVersionExpiration {
noncurrent_days: Some(1),
newer_noncurrent_versions: None,
}),
@@ -10355,12 +10379,12 @@ mod tests {
}
}
fn all_versions_expiration_lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn all_versions_expiration_lifecycle() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
days: Some(1),
expired_object_all_versions: Some(true),
..Default::default()
@@ -10379,29 +10403,29 @@ mod tests {
fn lock_enabled_without_default_retention() -> ObjectLockConfiguration {
ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: None,
}
}
fn lock_enabled_with_default_retention() -> ObjectLockConfiguration {
ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
days: Some(30),
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
mode: Some("COMPLIANCE".to_owned()),
years: None,
}),
}),
}
}
fn latest_transition_lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn latest_transition_lifecycle() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
@@ -10410,10 +10434,10 @@ mod tests {
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: Some(vec![Transition {
transitions: Some(vec![PersistedTransition {
days: Some(1),
date: None,
storage_class: Some(TransitionStorageClass::from_static("WARM")),
storage_class: Some("WARM".to_owned()),
}]),
}],
}
@@ -12950,8 +12974,9 @@ mod tests {
for (name, object, lock_config, expected) in cases {
let object_opts = lifecycle::object_opts_from_object_info(&object);
let evaluator_lock_config = lock_config.clone().map(Arc::new);
let batch_event = Evaluator::new(Arc::new(lifecycle.clone()))
.with_lock_retention(lock_config.clone().map(Arc::new))
.with_lock_retention(evaluator_lock_config)
.eval(&[object_opts])
.await
.expect("batch lifecycle evaluation should succeed")
@@ -14208,10 +14233,7 @@ mod tests {
let object = current_object_with_metadata(
ReplicationStatusType::Completed,
HashMap::from([
(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(),
),
(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()),
(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until),
]),
);
@@ -14223,7 +14245,7 @@ mod tests {
#[tokio::test]
async fn restored_copy_expiry_is_not_blocked_by_retention() {
let lifecycle = BucketLifecycleConfiguration {
let lifecycle = PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: Vec::new(),
};
@@ -14233,10 +14255,7 @@ mod tests {
let mut object = current_object_with_metadata(
ReplicationStatusType::Completed,
HashMap::from([
(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
ObjectLockRetentionMode::COMPLIANCE.to_string(),
),
(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()),
(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until),
]),
);
@@ -14453,7 +14472,7 @@ mod tests {
"removed" => {
metadata_sys::delete(&bucket, BUCKET_LIFECYCLE_CONFIG).await.unwrap();
}
"disabled" => changed.rules[0].status = ExpirationStatus::from_static(ExpirationStatus::DISABLED),
"disabled" => changed.rules[0].status = "Disabled".to_owned(),
"longer-days" => {
changed.rules[0]
.noncurrent_version_expiration
@@ -14569,7 +14588,7 @@ mod tests {
);
}
async fn install_test_expiry_policy(ecstore: &Arc<ECStore>, bucket: &str) -> Arc<BucketLifecycleConfiguration> {
async fn install_test_expiry_policy(ecstore: &Arc<ECStore>, bucket: &str) -> Arc<PersistedLifecycleConfiguration> {
let xml = br#"<LifecycleConfiguration><Rule><ID>expire</ID><Status>Enabled</Status><Filter><Prefix></Prefix></Filter><Expiration><Days>0</Days></Expiration><NoncurrentVersionExpiration><NoncurrentDays>0</NoncurrentDays></NoncurrentVersionExpiration></Rule></LifecycleConfiguration>"#;
metadata_sys::update_in(&ecstore.ctx, bucket, BUCKET_LIFECYCLE_CONFIG, xml.to_vec())
.await
@@ -15262,7 +15281,11 @@ mod tests {
#[tokio::test]
#[serial]
async fn tier_free_version_recovery_production_entrypoint_enqueues_seeded_item() {
let (disk_paths, ecstore) = setup_test_env().await;
let (disk_paths, ecstore) =
temp_env::async_with_vars([(super::ENV_TIER_FREE_VERSION_RECOVERY_ENABLED, Some("false"))], async {
setup_test_env().await
})
.await;
let tier = format!("RECOVERY{}", Uuid::new_v4().simple()).to_uppercase();
crate::services::tier::test_util::register_mock_tier(&ecstore.tier_config_mgr(), &tier).await;
let runtime_state = install_unconsumed_runtime_expiry_worker(&ecstore, 1).await;
+1 -1
View File
@@ -17,7 +17,7 @@ use crate::object_api::ObjectInfo;
pub use rustfs_lifecycle::{
Event, ExpirationOptions, IlmAction, LIFECYCLE_MALFORMED_XML_ERROR_KIND, Lifecycle, LifecycleCalculate, ObjectOpts,
RuleValidate, TRANSITION_COMPLETE, TRANSITION_PENDING, TransitionOptions, abort_incomplete_multipart_upload_due,
expected_expiry_time, expiration_action_has_valid_target,
expected_expiry_time, expiration_action_has_valid_target, format_lifecycle_date, parse_lifecycle_date,
};
pub fn object_opts_from_object_info(oi: &ObjectInfo) -> ObjectOpts {
@@ -14,7 +14,8 @@
use std::sync::Arc;
use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration};
use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
use rustfs_gateway_types::persistence::PersistedObjectLockConfiguration as ObjectLockConfiguration;
use time::OffsetDateTime;
use uuid::Uuid;
@@ -24,7 +25,7 @@ use crate::error::{Error, Result};
#[derive(Debug)]
pub(crate) struct LifecycleExpiryConfigs {
pub(crate) lifecycle: Option<Arc<BucketLifecycleConfiguration>>,
pub(crate) lifecycle: Option<Arc<PersistedLifecycleConfiguration>>,
pub(crate) object_lock: Option<Arc<ObjectLockConfiguration>>,
pub(crate) bucket_incarnation_id: Uuid,
pub(crate) table_bucket_enabled: bool,
@@ -100,7 +101,7 @@ pub(crate) async fn get_expiry_configs_for_options(
})
}
pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(PersistedLifecycleConfiguration, OffsetDateTime)> {
metadata_sys::get_lifecycle_config(bucket).await
}
@@ -110,15 +111,15 @@ mod tests {
use crate::bucket::metadata::BucketMetadata;
use crate::bucket::metadata_sys::{self, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::bucket::MakeBucketOptions;
use s3s::dto::{ExpirationStatus, LifecycleExpiration, LifecycleRule};
use rustfs_gateway_types::persistence::{PersistedLifecycleExpiration, PersistedLifecycleRule};
use serial_test::serial;
fn lifecycle_config() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn lifecycle_config() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
days: Some(1),
..Default::default()
}),
@@ -22,7 +22,7 @@ pub(crate) fn is_object_locked_by_metadata(user_defined: &HashMap<String, String
}
pub(crate) fn check_object_lock_for_deletion_with_config(
config: Option<&s3s::dto::ObjectLockConfiguration>,
config: Option<&rustfs_gateway_types::persistence::PersistedObjectLockConfiguration>,
obj_info: &ObjectInfo,
bypass_governance: bool,
) -> crate::error::Result<Option<ObjectLockBlockReason>> {
@@ -82,12 +82,13 @@ mod tests {
use crate::bucket::replication::{DeleteReplicationConfigSnapshot, ReplicationObjectBridge};
use crate::object_api::{ObjectInfo, ObjectOptions};
use crate::storage_api_contracts::object::ObjectToDelete;
use rustfs_scanner_metrics::metrics::IlmAction;
use s3s::dto::{
BucketVersioningStatus, DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication,
DeleteReplicationStatus, Destination, ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus,
VersioningConfiguration,
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use rustfs_gateway_types::persistence::{
PersistedOptionalReplicationStatus, PersistedReplicationConfiguration, PersistedReplicationDestination,
PersistedReplicationRule, PersistedReplicationStatus,
};
use rustfs_scanner_metrics::metrics::IlmAction;
use uuid::Uuid;
use super::*;
@@ -160,27 +161,27 @@ mod tests {
for purge_enabled in [false, true] {
let snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
delete_marker_replication: Some(DeleteMarkerReplication {
rules: vec![PersistedReplicationRule {
delete_marker_replication: Some(PersistedOptionalReplicationStatus {
status: Some(if marker_enabled {
DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)
Status::ENABLED.to_string()
} else {
DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)
Status::DISABLED.to_string()
}),
}),
delete_replication: Some(DeleteReplication {
delete_replication: Some(PersistedReplicationStatus {
status: if purge_enabled {
DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED)
Status::ENABLED.to_string()
} else {
DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED)
Status::DISABLED.to_string()
},
}),
destination: Destination {
destination: PersistedReplicationDestination {
bucket: "arn:rustfs:replication:target".to_string(),
..Default::default()
},
@@ -190,7 +191,7 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}],
}),
);
@@ -14,7 +14,7 @@
use std::sync::Arc;
use s3s::dto::BucketLifecycleConfiguration;
use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
use tokio::sync::RwLock;
use tokio_util::sync::CancellationToken;
@@ -52,7 +52,7 @@ pub(crate) fn deployment_id() -> Option<String> {
sources::deployment_id()
}
pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<BucketLifecycleConfiguration> {
pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<PersistedLifecycleConfiguration> {
sources::bucket_lifecycle_config(bucket).await
}
+51 -40
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use super::msgp_decode::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
use super::object_lock::{ObjectLockApi, ObjectLockStatusExt};
use super::object_lock::ObjectLockApi;
use super::versioning::VersioningApi;
use super::{quota::BucketQuota, target::BucketTargets};
use crate::bucket::replication::invalid_replication_config_status_field;
@@ -25,12 +25,17 @@ use crate::object_api::WriteCompletion;
use crate::runtime::sources as runtime_sources;
use crate::store::ECStore;
use byteorder::{BigEndian, ByteOrder, LittleEndian};
use rustfs_policy::policy::BucketPolicy;
use s3s::dto::{
AccelerateConfiguration, BucketLifecycleConfiguration, BucketLoggingStatus, BucketVersioningStatus, CORSConfiguration,
NotificationConfiguration, ObjectLockConfiguration, PublicAccessBlockConfiguration, ReplicationConfiguration,
RequestPaymentConfiguration, ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
use rustfs_gateway_types::cors_tagging::{PersistedCorsConfiguration as CORSConfiguration, PersistedTagging as Tagging};
use rustfs_gateway_types::persistence::{
PersistedAccelerateConfiguration as AccelerateConfiguration,
PersistedBucketEncryptionConfiguration as ServerSideEncryptionConfiguration,
PersistedBucketLoggingStatus as BucketLoggingStatus, PersistedLifecycleConfiguration,
PersistedNotificationConfiguration as NotificationConfiguration, PersistedObjectLockConfiguration as ObjectLockConfiguration,
PersistedPublicAccessBlockConfiguration as PublicAccessBlockConfiguration, PersistedReplicationConfiguration,
PersistedRequestPaymentConfiguration as RequestPaymentConfiguration,
PersistedVersioningConfiguration as VersioningConfiguration, PersistedWebsiteConfiguration as WebsiteConfiguration,
};
use rustfs_policy::policy::BucketPolicy;
use serde::Serializer;
use sha2::{Digest, Sha256};
use std::collections::HashMap;
@@ -40,23 +45,11 @@ use time::{Date, OffsetDateTime, PrimitiveDateTime, Time as CivilTime, UtcOffset
use tracing::error;
use uuid::Uuid;
// The serving-layer DTO impls for the storage-level Object Lock traits live
// here because this module owns the persisted `ObjectLockConfiguration`
// during the s3s ratchet migration (rustfs/backlog#1842).
// The persisted Object Lock configuration answers the storage-level trait
// here because this module owns the persisted `ObjectLockConfiguration`.
impl ObjectLockApi for ObjectLockConfiguration {
fn enabled(&self) -> bool {
self.object_lock_enabled
.as_ref()
.is_some_and(|v| v.as_str() == s3s::dto::ObjectLockEnabled::ENABLED)
}
}
impl ObjectLockStatusExt for s3s::dto::ObjectLockLegalHoldStatus {
fn valid(&self) -> bool {
matches!(
self.as_str(),
s3s::dto::ObjectLockLegalHoldStatus::ON | s3s::dto::ObjectLockLegalHoldStatus::OFF
)
self.object_lock_enabled.as_deref() == Some("Enabled")
}
}
@@ -439,6 +432,8 @@ pub struct BucketMetadata {
pub bucket_incarnation_id: Uuid,
pub(crate) bucket_incarnation_sidecar: bool,
pub(crate) bucket_creation_committed: bool,
// A legacy default is not proof of an interrupted creation.
pub(crate) bucket_creation_commit_record_present: bool,
pub policy_config_json: Vec<u8>,
pub notification_config_xml: Vec<u8>,
pub lifecycle_config_xml: Vec<u8>,
@@ -487,13 +482,13 @@ pub struct BucketMetadata {
pub policy_config: Option<BucketPolicy>,
pub notification_config: Option<NotificationConfiguration>,
pub lifecycle_config: Option<BucketLifecycleConfiguration>,
pub lifecycle_config: Option<PersistedLifecycleConfiguration>,
pub object_lock_config: Option<ObjectLockConfiguration>,
pub versioning_config: Option<VersioningConfiguration>,
pub sse_config: Option<ServerSideEncryptionConfiguration>,
pub tagging_config: Option<Tagging>,
pub quota_config: Option<BucketQuota>,
pub replication_config: Option<ReplicationConfiguration>,
pub replication_config: Option<PersistedReplicationConfiguration>,
pub bucket_target_config: Option<BucketTargets>,
pub bucket_target_config_meta: Option<HashMap<String, String>>,
pub cors_config: Option<CORSConfiguration>,
@@ -514,6 +509,7 @@ impl Default for BucketMetadata {
bucket_incarnation_id: Uuid::nil(),
bucket_incarnation_sidecar: false,
bucket_creation_committed: false,
bucket_creation_commit_record_present: false,
policy_config_json: Default::default(),
notification_config_xml: Default::default(),
lifecycle_config_xml: Default::default(),
@@ -752,7 +748,10 @@ impl BucketMetadata {
"TableBucketConfigUpdatedAt" => self.table_bucket_config_updated_at = read_msgp_time_value(rd)?,
"DurabilityConfigUpdatedAt" => self.durability_config_updated_at = read_msgp_time_value(rd)?,
"OnDemandMigrationConfigUpdatedAt" => self.on_demand_migration_config_updated_at = read_msgp_time_value(rd)?,
"BucketCreationCommitted" => self.bucket_creation_committed = read_msgp_bool(rd)?,
"BucketCreationCommitted" => {
self.bucket_creation_committed = read_msgp_bool(rd)?;
self.bucket_creation_commit_record_present = true;
}
other => {
tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
skip_msgp_value(rd)?;
@@ -765,8 +764,10 @@ impl BucketMetadata {
/// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
// Map size: MinIO fields (25) + RustFS extensions (22)
let map_len: u32 = 47;
// Preserve the absence of a legacy commit record across config rewrites.
let commit_record_present = self.bucket_creation_commit_record_present || self.bucket_creation_committed;
// Map size: MinIO fields (25) + RustFS extensions (21) + optional commit record.
let map_len: u32 = 46 + u32::from(commit_record_present);
rmp::encode::write_map_len(wr, map_len)?;
// MinIO field order (same as Go struct)
@@ -847,8 +848,10 @@ impl BucketMetadata {
write_msgp_time(wr, self.durability_config_updated_at)?;
rmp::encode::write_str(wr, "OnDemandMigrationConfigUpdatedAt")?;
write_msgp_time(wr, self.on_demand_migration_config_updated_at)?;
rmp::encode::write_str(wr, "BucketCreationCommitted")?;
rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
if commit_record_present {
rmp::encode::write_str(wr, "BucketCreationCommitted")?;
rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
}
Ok(())
}
@@ -1018,9 +1021,11 @@ impl BucketMetadata {
None
} else {
let config = deserialize::<VersioningConfiguration>(&data)?;
if config.status.as_ref().is_some_and(|status| {
!matches!(status.as_str(), BucketVersioningStatus::ENABLED | BucketVersioningStatus::SUSPENDED)
}) {
if config
.status
.as_ref()
.is_some_and(|status| !matches!(status.as_str(), "Enabled" | "Suspended"))
{
return Err(Error::other("bucket versioning configuration has an invalid status"));
}
Some(config)
@@ -1033,7 +1038,7 @@ impl BucketMetadata {
let config = if data.is_empty() {
None
} else {
let config = deserialize::<ReplicationConfiguration>(&data)?;
let config = deserialize::<PersistedReplicationConfiguration>(&data)?;
if let Some(field) = invalid_replication_config_status_field(&config) {
return Err(Error::other(format!("replication field {field} has an invalid status")));
}
@@ -1227,8 +1232,8 @@ impl BucketMetadata {
}
self.lifecycle_config = None;
if !self.lifecycle_config_xml.is_empty()
&& let Err(e) =
deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml).map(|c| self.lifecycle_config = Some(c))
&& let Err(e) = deserialize::<PersistedLifecycleConfiguration>(&self.lifecycle_config_xml)
.map(|c| self.lifecycle_config = Some(c))
{
tracing::warn!(
event = "bucket_metadata_parse_failed",
@@ -1314,8 +1319,8 @@ impl BucketMetadata {
}
self.replication_config = None;
if !self.replication_config_xml.is_empty()
&& let Err(e) =
deserialize::<ReplicationConfiguration>(&self.replication_config_xml).map(|c| self.replication_config = Some(c))
&& let Err(e) = deserialize::<PersistedReplicationConfiguration>(&self.replication_config_xml)
.map(|c| self.replication_config = Some(c))
{
tracing::warn!(
event = "bucket_metadata_parse_failed",
@@ -1695,6 +1700,13 @@ mod test {
let mut bm = BucketMetadata::unmarshal(&body[4..]).expect("unmarshal recovered blob");
assert_eq!(bm.name, "interop");
assert!(!bm.bucket_creation_committed, "legacy metadata must default to uncommitted");
assert!(!bm.bucket_creation_commit_record_present, "legacy metadata has no creation-commit record");
let rewritten = BucketMetadata::unmarshal(&bm.marshal_msg().expect("rewrite legacy metadata"))
.expect("read rewritten legacy metadata");
assert!(
!rewritten.bucket_creation_commit_record_present,
"rewriting legacy metadata must not fabricate a creation intent"
);
bm.parse_all_configs().expect("parse recovered configs");
assert!(bm.lifecycle_config.is_some());
}
@@ -1712,6 +1724,7 @@ mod test {
assert_eq!(bm.name, new.name);
assert!(new.bucket_creation_committed);
assert!(new.bucket_creation_commit_record_present);
assert!(!bm.bucket_incarnation_id.is_nil());
assert_eq!(bm.bucket_incarnation_id, new.bucket_incarnation_id);
}
@@ -1816,17 +1829,15 @@ mod test {
#[test]
fn object_locking_requires_lock_metadata_not_plain_versioning() {
use s3s::dto::ObjectLockEnabled;
let mut bm = BucketMetadata::new("test-bucket");
bm.versioning_config = Some(VersioningConfiguration {
status: Some(s3s::dto::BucketVersioningStatus::from_static("Enabled")),
status: Some("Enabled".to_owned()),
..Default::default()
});
assert!(!bm.object_locking());
bm.object_lock_config = Some(ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
..Default::default()
});
assert!(bm.object_locking());
+327 -114
View File
@@ -30,14 +30,18 @@ use crate::storage_api_contracts::heal::HealOperations as _;
use crate::storage_api_contracts::namespace::NamespaceLocking as _;
use crate::store::{ECStore, await_bucket_namespace_operation};
use futures::future::join_all;
use rustfs_gateway_types::cors_tagging::{PersistedCorsConfiguration as CORSConfiguration, PersistedTagging as Tagging};
use rustfs_gateway_types::persistence::{
PersistedAccelerateConfiguration as AccelerateConfiguration,
PersistedBucketEncryptionConfiguration as ServerSideEncryptionConfiguration,
PersistedBucketLoggingStatus as BucketLoggingStatus, PersistedLifecycleConfiguration,
PersistedNotificationConfiguration as NotificationConfiguration, PersistedObjectLockConfiguration as ObjectLockConfiguration,
PersistedPublicAccessBlockConfiguration as PublicAccessBlockConfiguration, PersistedReplicationConfiguration,
PersistedRequestPaymentConfiguration as RequestPaymentConfiguration,
PersistedVersioningConfiguration as VersioningConfiguration, PersistedWebsiteConfiguration as WebsiteConfiguration,
};
use rustfs_heal_contracts::heal_channel::HealOpts;
use rustfs_policy::policy::BucketPolicy;
use s3s::dto::ReplicationConfiguration;
use s3s::dto::{
AccelerateConfiguration, BucketLifecycleConfiguration, BucketLoggingStatus, CORSConfiguration, NotificationConfiguration,
ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRetentionMode, PublicAccessBlockConfiguration,
RequestPaymentConfiguration, ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
};
use std::collections::HashSet;
use std::time::Duration;
use std::{
@@ -168,7 +172,7 @@ pub(crate) fn object_lock_config_state_from_authoritative_metadata(bm: &BucketMe
if bm.lock_enabled {
return Ok(ObjectLockConfigState::Configured {
config: ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: None,
},
updated_at: bm.object_lock_config_updated_at,
@@ -212,17 +216,10 @@ pub(crate) fn configured_object_lock_state_for_tests(
) -> ObjectLockConfigState {
ObjectLockConfigState::Configured {
config: ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
rule: Some(s3s::dto::ObjectLockRule {
default_retention: Some(s3s::dto::DefaultRetention {
mode: Some(s3s::dto::ObjectLockRetentionMode::from_static(match mode {
crate::bucket::object_lock::types::RetentionMode::Governance => {
s3s::dto::ObjectLockRetentionMode::GOVERNANCE
}
crate::bucket::object_lock::types::RetentionMode::Compliance => {
s3s::dto::ObjectLockRetentionMode::COMPLIANCE
}
})),
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(rustfs_gateway_types::persistence::PersistedObjectLockRule {
default_retention: Some(rustfs_gateway_types::persistence::PersistedDefaultRetention {
mode: Some(mode.as_str().to_owned()),
days: Some(days),
years: None,
}),
@@ -233,7 +230,7 @@ pub(crate) fn configured_object_lock_state_for_tests(
}
fn validate_authoritative_object_lock_config(config: &ObjectLockConfiguration) -> Result<()> {
if config.object_lock_enabled.as_ref().map(ObjectLockEnabled::as_str) != Some(ObjectLockEnabled::ENABLED) {
if config.object_lock_enabled.as_deref() != Some("Enabled") {
return Err(Error::other("persisted bucket Object Lock enabled state is invalid"));
}
@@ -246,7 +243,7 @@ fn validate_authoritative_object_lock_config(config: &ObjectLockConfiguration) -
if !retention
.mode
.as_ref()
.is_some_and(|mode| matches!(mode.as_str(), ObjectLockRetentionMode::COMPLIANCE | ObjectLockRetentionMode::GOVERNANCE))
.is_some_and(|mode| matches!(mode.as_str(), "COMPLIANCE" | "GOVERNANCE"))
{
return Err(Error::other("persisted bucket Object Lock retention mode is invalid"));
}
@@ -683,7 +680,6 @@ pub(crate) async fn update_validated_in<F>(
bucket: &str,
config_file: &str,
data: Vec<u8>,
expected_incarnation_id: Option<Uuid>,
validate: F,
) -> Result<OffsetDateTime>
where
@@ -691,7 +687,7 @@ where
{
Box::pin(async {
let sys = require_bucket_metadata_sys_in(ctx)?;
let guard = acquire_config_write_guard_for_incarnation(sys.clone(), bucket, expected_incarnation_id).await?;
let guard = acquire_config_write_guard(sys.clone(), bucket).await?;
validate()?;
update_under_config_write_guard(sys, &guard, config_file, data, None).await
})
@@ -1396,7 +1392,7 @@ pub async fn get_public_access_block_config(bucket: &str) -> Result<(PublicAcces
bucket_meta_sys.get_public_access_block_config(bucket).await
}
pub async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
pub async fn get_lifecycle_config(bucket: &str) -> Result<(PersistedLifecycleConfiguration, OffsetDateTime)> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -1672,7 +1668,7 @@ pub(crate) async fn get_cached_quota_config_and_incarnation_in(
bucket_meta_sys.cached_quota_config_and_incarnation(bucket).await
}
pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
pub async fn get_replication_config(bucket: &str) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -1682,7 +1678,7 @@ pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfigur
pub(crate) async fn get_replication_config_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await;
@@ -1820,9 +1816,27 @@ pub(crate) fn current_local_meta_mutation_gen() -> u64 {
type LocalAuthorityCache = Arc<StdMutex<HashMap<String, (u64, Arc<BucketMetadata>)>>>;
#[derive(Debug, Clone)]
pub(crate) struct PendingBucketCreationProof {
pub(crate) generation: Uuid,
pub(crate) intent: Vec<u8>,
pub(crate) versioning_requested: bool,
pub(crate) witnesses: Vec<crate::disk::os::NativeBucketCreationWitness>,
}
impl PendingBucketCreationProof {
pub(crate) fn matches_metadata(&self, metadata: &BucketMetadata) -> Result<bool> {
Ok(!metadata.bucket_creation_committed
&& metadata.bucket_incarnation_sidecar
&& metadata.bucket_incarnation_id == self.generation
&& metadata.marshal_msg()? == self.intent)
}
}
#[derive(Debug, Clone)]
pub struct BucketMetadataSys {
metadata_map: Arc<RwLock<HashMap<String, Arc<BucketMetadata>>>>,
pending_bucket_creations: Arc<RwLock<HashMap<String, PendingBucketCreationProof>>>,
local_authority_cache: LocalAuthorityCache,
/// Serializes metadata-map commits and their derived cache updates for one
/// bucket. Namespace locks, when present, are acquired before this lock.
@@ -1867,6 +1881,7 @@ impl BucketMetadataSys {
pub fn new(api: Arc<ECStore>) -> Self {
Self {
metadata_map: Arc::new(RwLock::new(HashMap::new())),
pending_bucket_creations: Arc::new(RwLock::new(HashMap::new())),
local_authority_cache: Arc::new(StdMutex::new(HashMap::new())),
metadata_publish_locks: Arc::new(MetadataPublishLockRegistry {
locks: StdMutex::new(HashMap::new()),
@@ -1898,17 +1913,83 @@ impl BucketMetadataSys {
.expect("bucket metadata object store should still be live")
}
fn object_store_if_live(&self) -> Option<Arc<ECStore>> {
self.api.upgrade()
pub(crate) async fn pending_bucket_creation(&self, bucket: &str) -> Option<PendingBucketCreationProof> {
self.pending_bucket_creations.read().await.get(bucket).cloned()
}
/// Whether the backing `ECStore` has been dropped.
///
/// Used by `InstanceContext::clear_bucket_metadata_sys_if_stale` during
/// `ECStore::drop` so a shared bootstrap context does not retain a
/// `BucketMetadataSys` whose weak handle is dangling.
pub(crate) fn is_store_stale(&self) -> bool {
self.api.upgrade().is_none()
pub(crate) async fn grant_pending_bucket_creation(&self, bucket: &str, proof: PendingBucketCreationProof) {
self.pending_bucket_creations.write().await.insert(bucket.to_owned(), proof);
}
pub(crate) async fn revoke_pending_bucket_creation(&self, bucket: &str) {
self.pending_bucket_creations.write().await.remove(bucket);
}
pub(crate) async fn consume_pending_bucket_creation(
&self,
bucket: &str,
metadata: &BucketMetadata,
witnesses: &[crate::disk::os::NativeBucketCreationWitness],
) -> Result<bool> {
let mut pending = self.pending_bucket_creations.write().await;
let Some(proof) = pending.get(bucket) else {
return Ok(false);
};
if !proof.matches_metadata(metadata)?
|| proof.witnesses.len() != witnesses.len()
|| !proof
.witnesses
.iter()
.zip(witnesses)
.all(|(old, current)| old.matches(current))
|| !crate::disk::os::consume_native_creation_witnesses(witnesses)
{
return Ok(false);
}
pending.remove(bucket);
Ok(true)
}
#[cfg(test)]
pub(crate) async fn refresh_bucket_for_test(&self, bucket: &str) -> Result<()> {
let expected = self.metadata_map.read().await.get(bucket).cloned();
let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
self.load_bucket_under_namespace(bucket, MetadataLoadMode::Refresh, expected.as_ref(), &guard)
.await
}
#[cfg(test)]
pub(crate) async fn publish_persisted_metadata_for_test(&self, bucket: &str) -> Result<()> {
let expected = self.metadata_map.read().await.get(bucket).cloned();
let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
let (metadata, persisted) = await_bucket_namespace_operation(
Some(&guard),
bucket,
"test cached metadata load",
load_bucket_metadata_parse_with_presence(self.object_store(), bucket, true),
)
.await?;
self.publish_if_unchanged(bucket, expected.as_ref(), metadata, persisted, &guard)
.await
}
#[cfg(test)]
pub(crate) async fn migrate_bucket_creation_commit_for_test(&self, bucket: &str) -> Result<()> {
self.migrate_bucket_creation_commit(bucket).await.map(|_| ())
}
#[cfg(test)]
pub(crate) async fn legacy_migration_reports_missing_bucket_for_test(&self, bucket: &str) -> Result<bool> {
Ok(matches!(
self.migrate_legacy_metadata(bucket).await?,
BucketMetadataAuthority::MissingBucket
))
}
fn object_store_if_live(&self) -> Option<Arc<ECStore>> {
self.api.upgrade()
}
fn metadata_publish_lock(&self, bucket: &str) -> Arc<Mutex<MetadataPublishLockState>> {
@@ -2097,6 +2178,7 @@ impl BucketMetadataSys {
expected: Option<&Arc<BucketMetadata>>,
namespace_guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<()> {
self.revoke_pending_bucket_creation(bucket).await;
if !await_bucket_namespace_operation(
Some(namespace_guard),
bucket,
@@ -2180,6 +2262,7 @@ impl BucketMetadataSys {
persisted: bool,
namespace_guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<()> {
self.revoke_pending_bucket_creation(bucket).await;
if !persisted {
let _publish_guard = self
.lock_metadata_publish(bucket, namespace_guard, "refreshed bucket metadata absence publish")
@@ -2241,6 +2324,7 @@ impl BucketMetadataSys {
}
pub async fn set(&self, bucket: String, bm: Arc<BucketMetadata>) {
self.revoke_pending_bucket_creation(&bucket).await;
bump_local_meta_mutation_gen();
if !is_meta_bucketname(&bucket) {
let publish_lock = self.metadata_publish_lock(&bucket);
@@ -2264,6 +2348,7 @@ impl BucketMetadataSys {
if is_meta_bucketname(bucket) {
return false;
}
self.revoke_pending_bucket_creation(bucket).await;
let publish_lock = self.metadata_publish_lock(bucket);
let _publish_guard = publish_lock.lock().await;
let mut map = self.metadata_map.write().await;
@@ -2281,6 +2366,7 @@ impl BucketMetadataSys {
}
async fn _reset(&mut self) {
self.pending_bucket_creations.write().await.clear();
bump_local_meta_mutation_gen();
let mut map = self.metadata_map.write().await;
map.clear();
@@ -2420,6 +2506,7 @@ impl BucketMetadataSys {
/// server's metadata never leaks into the ambient (first) instance.
pub(crate) async fn persist_and_set(&self, bm: BucketMetadata) -> Result<()> {
let mut bm = bm;
self.revoke_pending_bucket_creation(&bm.name).await;
bm.save_with_store(self.object_store()).await?;
self.set(bm.name.clone(), Arc::new(bm)).await;
@@ -2428,6 +2515,7 @@ impl BucketMetadataSys {
}
async fn persist_new_and_set(&self, mut bm: BucketMetadata) -> Result<()> {
self.revoke_pending_bucket_creation(&bm.name).await;
bm.bucket_creation_committed = true;
bm.save_with_store_committed(self.object_store()).await?;
save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
@@ -2437,7 +2525,9 @@ impl BucketMetadataSys {
}
async fn persist_new_bucket_metadata_intent(&self, mut bm: BucketMetadata) -> Result<()> {
self.revoke_pending_bucket_creation(&bm.name).await;
bm.bucket_creation_committed = false;
bm.bucket_creation_commit_record_present = true;
bm.save_with_store(self.object_store()).await?;
save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
bm.bucket_incarnation_sidecar = true;
@@ -2447,6 +2537,7 @@ impl BucketMetadataSys {
}
async fn commit_bucket_metadata(&self, mut bm: BucketMetadata) -> Result<()> {
self.revoke_pending_bucket_creation(&bm.name).await;
bm.parse_all_configs()?;
bm.bucket_incarnation_sidecar = true;
// The caller holds the full bucket creation fence (lifecycle, metadata
@@ -2495,6 +2586,7 @@ impl BucketMetadataSys {
bucket: &str,
namespace_guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<()> {
self.revoke_pending_bucket_creation(bucket).await;
let expected = self.metadata_map.read().await.get(bucket).cloned();
if !self
.bucket_exists(bucket, namespace_guard, "peer bucket metadata existence check")
@@ -2551,6 +2643,7 @@ impl BucketMetadataSys {
#[cfg(test)]
self.lazy_disk_loads.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
self.revoke_pending_bucket_creation(bucket).await;
let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
#[cfg(test)]
@@ -2876,6 +2969,7 @@ impl BucketMetadataSys {
/// revalidated at write quorum, so a stale snapshot can neither revert an
/// acknowledged configuration update nor outlive a delete/recreate.
async fn migrate_bucket_creation_commit(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
self.revoke_pending_bucket_creation(bucket).await;
let transaction_lock = self
.object_store()
.new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
@@ -2953,6 +3047,7 @@ impl BucketMetadataSys {
&self,
bucket: &str,
) -> Result<BucketMetadataAuthority> {
self.revoke_pending_bucket_creation(bucket).await;
#[cfg(test)]
if self.object_lock_disk_read_errors.write().await.remove(bucket) {
return Err(Error::other(format!("injected Object Lock metadata disk read failure: {bucket}")));
@@ -3175,7 +3270,7 @@ impl BucketMetadataSys {
}
}
pub async fn get_lifecycle_config(&self, bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
pub async fn get_lifecycle_config(&self, bucket: &str) -> Result<(PersistedLifecycleConfiguration, OffsetDateTime)> {
let (bm, _) = self.get_config(bucket).await?;
match ConfigState::of(&bm.lifecycle_config_xml, &bm.lifecycle_config).require(bucket, BUCKET_LIFECYCLE_CONFIG)? {
@@ -3298,7 +3393,7 @@ impl BucketMetadataSys {
}))
}
pub async fn get_replication_config(&self, bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
pub async fn get_replication_config(&self, bucket: &str) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
let (bm, _) = self.get_config(bucket).await?;
if !bm.replication_config_xml.is_empty() && bm.replication_config.is_none() {
@@ -3400,6 +3495,7 @@ mod tests {
};
use crate::bucket::target::{BucketTarget, BucketTargetType, Credentials};
use crate::config::com::read_config;
use crate::disk::{DiskAPI as _, RUSTFS_META_TMP_BUCKET};
use crate::storage_api_contracts::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
use byteorder::{ByteOrder as _, LittleEndian};
use serial_test::serial;
@@ -3454,6 +3550,180 @@ mod tests {
),
];
#[tokio::test]
#[serial(storage_class_env)]
async fn persistence_entry_failures_revoke_pending_creation_before_cache_publish() {
let (dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let sys = bucket_metadata_sys_of(&store.ctx).expect("instance metadata system");
let sys = sys.read().await.clone();
let options = MakeBucketOptions {
lock_enabled: true,
..Default::default()
};
for entry in ["save", "persist", "persist-new", "commit"] {
let bucket = format!("revoke-{entry}-{}", Uuid::new_v4());
store.fail_next_bucket_creation_after_intent_for_test(&bucket);
store
.make_bucket(&bucket, &options)
.await
.expect_err("stop the real producer before physical publication");
sys.pending_bucket_creation(&bucket)
.await
.expect("the producer must leave a real live creation permission");
let (intent, persisted) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the original producer intent");
assert!(persisted, "{entry}: the complete original intent must be persisted");
let original_intent = intent.marshal_msg().expect("encode the original intent");
// Keep the creation fence order used by the real producer.
let lifecycle_guard = store
.acquire_bucket_lifecycle_write_lock(&bucket)
.await
.expect("hold the bucket lifecycle fence");
let transaction_guard = acquire_bucket_metadata_transaction_lock_in(&store.ctx, &bucket)
.await
.expect("hold the metadata transaction fence");
let namespace_lock = store
.new_ns_lock(&bucket, &bucket)
.await
.expect("create the exact bucket namespace lock");
let namespace_guard = namespace_lock
.get_write_lock(crate::set_disk::get_lock_acquire_timeout())
.await
.expect("hold the exact bucket namespace fence");
// A file at each native staging root causes actual IO to fail before
// metadata can reach the cache-publishing set call. Preserve the
// original staging directories so no fixture data is discarded.
for dir in &dirs {
let staging = dir.path().join(RUSTFS_META_TMP_BUCKET);
let saved = dir.path().join(".rustfs.sys/revocation-saved-tmp");
std::fs::rename(&staging, &saved).expect("preserve the native staging directory");
std::fs::write(&staging, b"block metadata staging with a non-directory")
.expect("install an actual non-directory IO obstruction");
}
let result = match entry {
"save" => sys.save(intent).await,
"persist" => sys.persist_and_set(intent).await,
"persist-new" => sys.persist_new_and_set(intent).await,
"commit" => sys.commit_bucket_metadata(intent).await,
_ => unreachable!(),
};
for dir in &dirs {
let staging = dir.path().join(RUSTFS_META_TMP_BUCKET);
let saved = dir.path().join(".rustfs.sys/revocation-saved-tmp");
std::fs::remove_file(&staging).expect("remove only the fixture's staging obstruction");
std::fs::rename(&saved, &staging).expect("restore the complete original staging directory");
}
drop((namespace_guard, transaction_guard, lifecycle_guard));
result.expect_err("native persistence must fail before its downstream cache set");
assert!(
sys.pending_bucket_creation(&bucket).await.is_none(),
"{entry}: the entry revoker must retire live permission even when persistence fails"
);
let (after_failure, present) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the intent after the failed persistence entry");
assert!(present, "{entry}: a failed persistence entry must retain the original metadata");
assert_eq!(
after_failure
.marshal_msg()
.expect("encode the original intent after persistence failure"),
original_intent,
"{entry}: native IO failure must not change the original generation or complete intent"
);
let error = store
.make_bucket(&bucket, &options)
.await
.expect_err("a failed ordinary persistence entry cannot preserve creation permission");
assert!(error.to_string().contains("administrator recovery"), "{entry}: {error}");
let (after_retry, _) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the original intent after the rejected retry");
assert_eq!(
after_retry.marshal_msg().expect("encode the intent after the rejected retry"),
original_intent,
"{entry}: the rejected retry must preserve the complete original intent"
);
for disk in store.pools[0].disk_set[0].disks.read().await.iter() {
assert!(
matches!(
disk.as_ref().expect("native disk").stat_volume(&bucket).await,
Err(crate::disk::error::DiskError::VolumeNotFound)
),
"{entry}: failed persistence and the rejected retry must not publish a physical volume"
);
}
}
}
#[tokio::test]
#[serial(storage_class_env)]
async fn metadata_reset_revokes_real_pending_creation_without_changing_persisted_intent() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = format!("revoke-reset-{}", Uuid::new_v4());
let options = MakeBucketOptions {
lock_enabled: true,
..Default::default()
};
store.fail_next_bucket_creation_after_intent_for_test(&bucket);
store
.make_bucket(&bucket, &options)
.await
.expect_err("stop the real producer before physical publication");
let sys = bucket_metadata_sys_of(&store.ctx).expect("instance metadata system");
let mut sys = sys.read().await.clone();
sys.pending_bucket_creation(&bucket)
.await
.expect("the real producer must leave live creation permission before reset");
let (intent, persisted) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the original intent before cache reset");
assert!(persisted, "cache reset starts from a complete real persisted intent");
let original_intent = intent.marshal_msg().expect("encode the original reset intent");
sys._reset().await;
assert!(
sys.pending_bucket_creation(&bucket).await.is_none(),
"reset must revoke the producer-issued permission shared with the instance metadata system"
);
let (after_reset, present) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the original intent after cache reset");
assert!(present, "cache reset must retain the persisted intent");
assert_eq!(
after_reset.marshal_msg().expect("encode the intent after cache reset"),
original_intent,
"cache reset must leave the original generation and complete persisted intent unchanged"
);
let error = store
.make_bucket(&bucket, &options)
.await
.expect_err("reset cannot regrant the producer's old creation permission");
assert!(error.to_string().contains("administrator recovery"), "{error}");
let (after_retry, _) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
.await
.expect("read the intent after the reset-induced retry rejection");
assert_eq!(
after_retry
.marshal_msg()
.expect("encode the intent after the rejected reset retry"),
original_intent,
"the reset-induced rejection must preserve the original complete intent"
);
for disk in store.pools[0].disk_set[0].disks.read().await.iter() {
assert!(
matches!(
disk.as_ref().expect("native disk").stat_volume(&bucket).await,
Err(crate::disk::error::DiskError::VolumeNotFound)
),
"cache reset and the rejected retry must not publish a physical volume"
);
}
}
#[tokio::test]
async fn g_d3_003_new_writer_replication_loads_without_fail_closed_state() {
let (dirs, store) = isolated_store_over_temp_disks().await;
@@ -3642,7 +3912,7 @@ mod tests {
let mut versioned = BucketMetadata::new("authoritative-versioned");
versioned.versioning_config_xml = b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec();
versioned.versioning_config = Some(VersioningConfiguration {
status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
});
sys.set("authoritative-versioned".to_string(), Arc::new(versioned)).await;
@@ -3765,25 +4035,24 @@ mod tests {
else {
panic!("legacy lock-enabled metadata must remain configured");
};
assert_eq!(
config.object_lock_enabled.as_ref().map(|value| value.as_str()),
Some(ObjectLockEnabled::ENABLED)
);
assert_eq!(config.object_lock_enabled.as_deref(), Some("Enabled"));
assert!(config.rule.is_none());
}
#[test]
fn authoritative_object_lock_state_rejects_invalid_retention_periods() {
use s3s::dto::{DefaultRetention, ObjectLockRule};
use rustfs_gateway_types::persistence::{
PersistedDefaultRetention as DefaultRetention, PersistedObjectLockRule as ObjectLockRule,
};
for (days, years) in [(Some(0), None), (Some(-1), None), (Some(1), Some(1)), (None, None)] {
let mut metadata = BucketMetadata::new("invalid-object-lock");
metadata.object_lock_config = Some(ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
days,
mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
mode: Some("COMPLIANCE".to_owned()),
years,
}),
}),
@@ -3798,9 +4067,11 @@ mod tests {
#[test]
fn authoritative_object_lock_state_rejects_invalid_enabled_mode_and_rule() {
use s3s::dto::{DefaultRetention, ObjectLockRule};
use rustfs_gateway_types::persistence::{
PersistedDefaultRetention as DefaultRetention, PersistedObjectLockRule as ObjectLockRule,
};
for enabled in [None, Some(ObjectLockEnabled::from("Disabled".to_string()))] {
for enabled in [None, Some("Disabled".to_string())] {
let mut metadata = BucketMetadata::new("invalid-object-lock-enabled");
metadata.lock_enabled = true;
metadata.object_lock_config = Some(ObjectLockConfiguration {
@@ -3812,11 +4083,11 @@ mod tests {
let mut invalid_mode = BucketMetadata::new("invalid-object-lock-mode");
invalid_mode.object_lock_config = Some(ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(ObjectLockRule {
default_retention: Some(DefaultRetention {
days: Some(1),
mode: Some(ObjectLockRetentionMode::from("invalid".to_string())),
mode: Some("invalid".to_string()),
years: None,
}),
}),
@@ -3825,7 +4096,7 @@ mod tests {
let mut missing_retention = BucketMetadata::new("missing-object-lock-retention");
missing_retention.object_lock_config = Some(ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(ObjectLockRule { default_retention: None }),
});
assert!(object_lock_config_state_from_authoritative_metadata(&missing_retention).is_err());
@@ -3835,7 +4106,7 @@ mod tests {
fn object_lock_ever_enabled_marker_survives_config_removal() {
let mut metadata = BucketMetadata::new("lock-config-removed");
let config = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: None,
};
metadata
@@ -3853,10 +4124,7 @@ mod tests {
else {
panic!("an ever-enabled bucket must remain Object Lock configured");
};
assert_eq!(
config.object_lock_enabled.as_ref().map(ObjectLockEnabled::as_str),
Some(ObjectLockEnabled::ENABLED)
);
assert_eq!(config.object_lock_enabled.as_deref(), Some("Enabled"));
}
#[tokio::test]
@@ -3872,7 +4140,7 @@ mod tests {
let node = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore.clone())));
let config = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: None,
};
update_with_sys(
@@ -4853,7 +5121,7 @@ mod tests {
#[tokio::test]
async fn update_config_with_persists_tagging_rewrite_across_disk_reload() {
use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
use s3s::dto::Tag;
use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
@@ -5017,7 +5285,7 @@ mod tests {
#[tokio::test]
async fn concurrent_update_config_with_calls_do_not_lose_writes() {
use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
use s3s::dto::Tag;
use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
let sys = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore)));
@@ -5117,7 +5385,7 @@ mod tests {
let mut persisted = BucketMetadata::new("reload-bucket");
persisted.policy_config_json = b"persisted-marker".to_vec();
let lock_config = ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
rule: None,
};
persisted
@@ -5210,61 +5478,6 @@ mod tests {
);
}
#[tokio::test]
#[serial]
async fn validated_config_update_rejects_a_different_bucket_incarnation() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "catalog-cleanup-incarnation";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let incarnation = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("read bucket incarnation");
let called = std::sync::atomic::AtomicBool::new(false);
let result = store
.update_bucket_metadata_config_for_incarnation_validated(
bucket,
"quota.json",
br#"{"quota":1}"#.to_vec(),
Uuid::new_v4(),
|| {
called.store(true, std::sync::atomic::Ordering::SeqCst);
Ok(())
},
)
.await;
assert!(matches!(result, Err(Error::BucketNotFound(name)) if name == bucket));
assert!(
!called.load(std::sync::atomic::Ordering::SeqCst),
"incarnation must be checked before the external fence"
);
let (metadata, _) = get_config_from_disk_with_presence_in(&store.ctx, bucket)
.await
.expect("read unchanged bucket metadata");
assert!(
metadata.quota_config_json.is_empty(),
"a stale cleanup must not modify the replacement bucket"
);
store
.update_bucket_metadata_config_for_incarnation_validated(
bucket,
"quota.json",
br#"{"quota":1}"#.to_vec(),
incarnation,
|| Ok(()),
)
.await
.expect("the current incarnation permits the validated update");
let (metadata, _) = get_config_from_disk_with_presence_in(&store.ctx, bucket)
.await
.expect("read updated bucket metadata");
assert_eq!(metadata.quota_config_json, br#"{"quota":1}"#);
}
#[tokio::test]
#[serial]
async fn quota_admission_rejects_stale_cached_no_quota() {
@@ -5451,7 +5664,7 @@ mod tests {
#[serial]
async fn bucket_delete_waits_for_config_mutation_fence() {
use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
use s3s::dto::Tag;
use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
+2 -9
View File
@@ -17,15 +17,8 @@ pub mod objectlock_sys;
pub mod types;
/// Whether a bucket Object Lock configuration has locking enabled. The
/// serving-layer `ObjectLockConfiguration` DTO implements this in
/// the bucket-metadata module, which owns the persisted configuration type
/// during the s3s ratchet migration (rustfs/backlog#1842).
/// persisted `ObjectLockConfiguration` implements this in the bucket-metadata
/// module, which owns the persisted configuration type.
pub trait ObjectLockApi {
fn enabled(&self) -> bool;
}
/// Whether a legal-hold status value is one of the two valid wire values.
/// Implemented for the serving-layer DTO in the bucket-metadata module.
pub trait ObjectLockStatusExt {
fn valid(&self) -> bool;
}
@@ -12,6 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
pub use rustfs_replication::config_cache_serde;
pub use rustfs_replication::{
ObjectOpts, OperatorRuleContract, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_READ_ONLY_HISTORICAL_FIELDS,
REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, REPLICATION_CAPABILITY_CONTRACT_VERSION,
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use s3s::dto::ReplicationConfiguration;
use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
use super::replication_config_boundary::ReplicationConfigurationExt as _;
use super::replication_filemeta_boundary::{
@@ -42,7 +42,7 @@ impl ReplicationLifecycleBridge {
dead_code,
reason = "declared boundary surface for the ECStore replication split plan; no caller in this port (backlog#1823)"
)]
pub(crate) fn new_config(config: ReplicationConfiguration) -> ReplicationLifecycleConfig {
pub(crate) fn new_config(config: PersistedReplicationConfiguration) -> ReplicationLifecycleConfig {
ReplicationConfig::new(Some(config), None)
}
@@ -118,16 +118,17 @@ impl ReplicationLifecycleBridge {
#[cfg(test)]
mod tests {
use s3s::dto::{Destination, ReplicationRule, ReplicationRuleStatus};
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::{PersistedReplicationDestination, PersistedReplicationRule};
use super::super::replication_filemeta_boundary::{ReplicateTargetDecision, VersionPurgeStatusType};
use super::*;
fn replication_rule() -> ReplicationRule {
ReplicationRule {
fn replication_rule() -> PersistedReplicationRule {
PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: None,
destination: Destination {
destination: PersistedReplicationDestination {
bucket: "arn:aws:s3:::target-bucket".to_string(),
..Default::default()
},
@@ -137,13 +138,13 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}
}
#[test]
fn has_pending_version_purge_preserves_replication_active_rule_behavior() {
let config = ReplicationLifecycleBridge::new_config(ReplicationConfiguration {
let config = ReplicationLifecycleBridge::new_config(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![replication_rule()],
});
@@ -17,8 +17,8 @@ use std::sync::Arc;
use crate::bucket::{metadata::BucketMetadata, metadata_sys};
use crate::disk::{BUCKET_META_PREFIX, RUSTFS_META_BUCKET};
use crate::runtime::instance::InstanceContext;
use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
use rustfs_utils::path::path_join_buf;
use s3s::dto::ReplicationConfiguration;
use time::OffsetDateTime;
use super::replication_error_boundary::{Error, Result};
@@ -36,11 +36,11 @@ impl ReplicationMetadataStore {
pub(crate) const FORCE_DELETE_REPLICATION_FILE: &'static str = "config/replication/force-delete.bin";
pub(crate) const FORCE_DELETE_REPLICATION_TRANSACTION_LOCK: &'static str = "config/replication/force-delete.bin.transaction";
pub(crate) async fn replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
pub(crate) async fn replication_config(bucket: &str) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
metadata_sys::get_replication_config(bucket).await
}
pub(crate) async fn optional_replication_config(bucket: &str) -> Result<Option<ReplicationConfiguration>> {
pub(crate) async fn optional_replication_config(bucket: &str) -> Result<Option<PersistedReplicationConfiguration>> {
let config = match Self::replication_config(bucket).await {
Ok((config, _)) => Some(config),
Err(err) => {
@@ -16,7 +16,9 @@ use std::{collections::HashMap, fmt, sync::Arc};
use super::replication_filemeta_boundary::metadata_keys;
use crate::bucket::metadata::BucketMetadata;
use s3s::dto::{BucketVersioningStatus, ReplicationConfiguration, ReplicationRuleStatus, VersioningConfiguration};
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use serde::{Deserialize, Serialize};
use time::OffsetDateTime;
use tracing::error;
@@ -39,7 +41,7 @@ use super::replication_target_boundary::{BucketTargets, ReplicationTargetStore};
use super::replication_versioning_boundary::ReplicationVersioningStore;
use super::runtime_boundary as runtime_sources;
pub(crate) async fn get_replication_config(bucket: &str) -> Result<Option<ReplicationConfiguration>> {
pub(crate) async fn get_replication_config(bucket: &str) -> Result<Option<PersistedReplicationConfiguration>> {
let config = ReplicationMetadataStore::optional_replication_config(bucket).await?;
validate_delete_replication_config(&VersioningConfiguration::default(), config.as_ref())?;
Ok(config)
@@ -64,7 +66,7 @@ impl DeleteReplicationConfigSnapshot {
#[cfg(test)]
pub(crate) fn from_configs_for_test(
versioning: VersioningConfiguration,
replication: Option<ReplicationConfiguration>,
replication: Option<PersistedReplicationConfiguration>,
) -> Self {
let metadata = replication.map(|config| {
let mut metadata = BucketMetadata::new("test-bucket");
@@ -78,7 +80,7 @@ impl DeleteReplicationConfigSnapshot {
&self.versioning
}
pub fn replication_config(&self) -> Option<&ReplicationConfiguration> {
pub fn replication_config(&self) -> Option<&PersistedReplicationConfiguration> {
self.metadata
.as_ref()
.and_then(|metadata| metadata.replication_config.as_ref())
@@ -101,7 +103,7 @@ impl DeleteReplicationConfigSnapshot {
pub(crate) fn active_delete_marker_rules_require_tags(&self, object: &str) -> bool {
self.replication_config().is_some_and(|config| {
config.rules.iter().any(|rule| {
if rule.status == ReplicationRuleStatus::from_static(ReplicationRuleStatus::DISABLED) {
if rule.status == Status::DISABLED.as_str() {
return false;
}
if !object.starts_with(rule.prefix()) {
@@ -122,12 +124,12 @@ impl DeleteReplicationConfigSnapshot {
fn validate_delete_replication_config(
versioning: &VersioningConfiguration,
config: Option<&ReplicationConfiguration>,
config: Option<&PersistedReplicationConfiguration>,
) -> Result<()> {
if versioning
.status
.as_ref()
.is_some_and(|status| !matches!(status.as_str(), BucketVersioningStatus::ENABLED | BucketVersioningStatus::SUSPENDED))
.is_some_and(|status| !matches!(status.as_str(), "Enabled" | "Suspended"))
{
return Err(super::replication_error_boundary::Error::other(
"bucket versioning configuration has an invalid status",
@@ -144,7 +146,7 @@ fn validate_delete_replication_config(
let role = config.role.trim();
let mut role_destination = None;
for rule in &config.rules {
if rule.status.as_str() == ReplicationRuleStatus::ENABLED {
if rule.status.as_str() == Status::ENABLED.as_str() {
let destination = rule.destination.bucket.trim();
if role.is_empty() && destination.is_empty() {
return Err(super::replication_error_boundary::Error::other(
@@ -169,7 +171,7 @@ fn validate_delete_replication_config(
Ok(())
}
fn replication_config_from_metadata(metadata: &BucketMetadata) -> Result<Option<&ReplicationConfiguration>> {
fn replication_config_from_metadata(metadata: &BucketMetadata) -> Result<Option<&PersistedReplicationConfiguration>> {
if !metadata.replication_config_xml.is_empty() && metadata.replication_config.is_none() {
return Err(super::replication_error_boundary::Error::other(
"persisted bucket replication configuration is invalid",
@@ -250,12 +252,15 @@ pub(crate) async fn load_delete_replication_config_in(
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct ReplicationConfig {
pub config: Option<ReplicationConfiguration>,
/// Carried in the scanner's persisted data usage cache, so its serde form
/// is the one the s3s DTO wrote there (see `config_cache_serde` in the replication crate).
#[serde(with = "super::replication_config_boundary::config_cache_serde")]
pub config: Option<PersistedReplicationConfiguration>,
pub remotes: Option<BucketTargets>,
}
impl ReplicationConfig {
pub fn new(config: Option<ReplicationConfiguration>, remotes: Option<BucketTargets>) -> Self {
pub fn new(config: Option<PersistedReplicationConfiguration>, remotes: Option<BucketTargets>) -> Self {
Self { config, remotes }
}
@@ -468,7 +473,7 @@ fn check_replicate_delete_with_config(
oi: &ObjectInfo,
del_opts: &ObjectOptions,
source_error: bool,
config: Option<&ReplicationConfiguration>,
config: Option<&PersistedReplicationConfiguration>,
existing_delete_marker: bool,
trust_persisted_replica_status: bool,
) -> ReplicateDecision {
@@ -587,7 +592,7 @@ pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplic
}
fn metadata_target_should_replicate(
cfg: &ReplicationConfiguration,
cfg: &PersistedReplicationConfiguration,
opts: &ObjectOpts,
mopts: &MustReplicateOptions,
arn: &str,
@@ -597,20 +602,21 @@ fn metadata_target_should_replicate(
#[cfg(test)]
mod tests {
use s3s::dto::{
DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication, DeleteReplicationStatus, Destination,
ReplicaModifications, ReplicationRule, ReplicationRuleFilter, ReplicationRuleStatus, SourceSelectionCriteria, Tag,
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::{
PersistedOptionalReplicationStatus, PersistedReplicationDestination, PersistedReplicationFilter,
PersistedReplicationRule, PersistedReplicationStatus, PersistedReplicationTag, PersistedSourceSelectionCriteria,
};
use super::super::replication_filemeta_boundary::VersionPurgeStatusType;
use super::super::replication_target_boundary::BucketTarget;
use super::*;
fn replication_rule() -> ReplicationRule {
ReplicationRule {
fn replication_rule() -> PersistedReplicationRule {
PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: None,
destination: Destination {
destination: PersistedReplicationDestination {
bucket: "arn:aws:s3:::target-bucket".to_string(),
..Default::default()
},
@@ -620,7 +626,7 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}
}
@@ -629,11 +635,11 @@ mod tests {
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let mut rule = replication_rule();
rule.destination.bucket = arn.to_string();
rule.filter = Some(ReplicationRuleFilter {
rule.filter = Some(PersistedReplicationFilter {
prefix: Some("admitted/".to_string()),
..Default::default()
});
let cfg = ReplicationConfiguration {
let cfg = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![rule],
};
@@ -671,7 +677,7 @@ mod tests {
assert!(!empty.replicate(&ObjectOpts::default()));
let config = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![replication_rule()],
}),
@@ -725,7 +731,7 @@ mod tests {
fn delete_snapshot_rejects_enabled_rules_without_a_destination() {
let mut rule = replication_rule();
rule.destination.bucket.clear();
let config = ReplicationConfiguration {
let config = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![rule],
};
@@ -741,7 +747,7 @@ mod tests {
let first = replication_rule();
let mut second = replication_rule();
second.destination.bucket = "arn:aws:s3:::other-target".to_string();
let config = ReplicationConfiguration {
let config = PersistedReplicationConfiguration {
role: "arn:aws:s3:::role-target".to_string(),
rules: vec![first, second],
};
@@ -752,47 +758,47 @@ mod tests {
#[test]
fn delete_snapshot_rejects_unknown_status_values() {
let invalid_versioning = VersioningConfiguration {
status: Some("Enabld".to_string().into()),
status: Some("Enabld".to_string()),
..Default::default()
};
assert!(validate_delete_replication_config(&invalid_versioning, None).is_err());
let mut invalid_rule = replication_rule();
invalid_rule.status = "Enabld".to_string().into();
let config = ReplicationConfiguration {
invalid_rule.status = "Enabld".to_string();
let config = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![invalid_rule],
};
assert!(validate_delete_replication_config(&VersioningConfiguration::default(), Some(&config)).is_err());
let mut invalid_delete = replication_rule();
invalid_delete.delete_replication = Some(DeleteReplication {
status: "Enabld".to_string().into(),
invalid_delete.delete_replication = Some(PersistedReplicationStatus {
status: "Enabld".to_string(),
});
let config = ReplicationConfiguration {
let config = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![invalid_delete],
};
assert!(validate_delete_replication_config(&VersioningConfiguration::default(), Some(&config)).is_err());
let mut invalid_delete_marker = replication_rule();
invalid_delete_marker.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some("Enabld".to_string().into()),
invalid_delete_marker.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
status: Some("Enabld".to_string()),
});
let config = ReplicationConfiguration {
let config = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![invalid_delete_marker],
};
assert!(validate_delete_replication_config(&VersioningConfiguration::default(), Some(&config)).is_err());
let mut invalid_replica_modifications = replication_rule();
invalid_replica_modifications.source_selection_criteria = Some(SourceSelectionCriteria {
replica_modifications: Some(ReplicaModifications {
status: "Enabld".to_string().into(),
invalid_replica_modifications.source_selection_criteria = Some(PersistedSourceSelectionCriteria {
replica_modifications: Some(PersistedReplicationStatus {
status: "Enabld".to_string(),
}),
sse_kms_encrypted_objects: None,
});
let config = ReplicationConfiguration {
let config = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![invalid_replica_modifications],
};
@@ -804,11 +810,11 @@ mod tests {
let mut metadata = BucketMetadata::new("bucket");
metadata.versioning_config_xml = b"configured".to_vec();
metadata.versioning_config = Some(VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
});
metadata.replication_config_xml = b"configured".to_vec();
metadata.replication_config = Some(ReplicationConfiguration {
metadata.replication_config = Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![replication_rule()],
});
@@ -848,7 +854,7 @@ mod tests {
);
let mut inconsistent = BucketMetadata::new("bucket");
inconsistent.replication_config = Some(ReplicationConfiguration {
inconsistent.replication_config = Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![replication_rule()],
});
@@ -863,11 +869,11 @@ mod tests {
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let mut rule = replication_rule();
rule.destination.bucket = arn.to_string();
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
status: Some(Status::ENABLED.to_string()),
});
let mut metadata = BucketMetadata::new("bucket");
metadata.replication_config = Some(ReplicationConfiguration {
metadata.replication_config = Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![rule],
});
@@ -898,27 +904,27 @@ mod tests {
fn delete_marker_source_read_is_required_only_for_tag_filtered_rules() {
let mut prefix_rule = replication_rule();
prefix_rule.prefix = Some("logs/".to_string());
prefix_rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
prefix_rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
status: Some(Status::ENABLED.to_string()),
});
let prefix_snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![prefix_rule],
}),
);
let mut tag_rule = replication_rule();
tag_rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
tag_rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
status: Some(Status::ENABLED.to_string()),
});
tag_rule.filter = Some(ReplicationRuleFilter {
tag: Some(Tag {
tag_rule.filter = Some(PersistedReplicationFilter {
tag: Some(PersistedReplicationTag {
key: Some("class".to_string()),
value: Some("audit".to_string()),
}),
@@ -926,10 +932,10 @@ mod tests {
});
let tag_snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![tag_rule],
}),
@@ -944,14 +950,14 @@ mod tests {
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let mut rule = replication_rule();
rule.destination.bucket = arn.to_string();
rule.delete_replication = Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
rule.delete_replication = Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
});
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)),
rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
status: Some(Status::DISABLED.to_string()),
});
let config = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![rule],
}),
@@ -985,22 +991,22 @@ mod tests {
#[test]
fn live_delete_does_not_trust_persisted_replica_status() {
let mut rule = replication_rule();
rule.delete_replication = Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
rule.delete_replication = Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
});
rule.source_selection_criteria = Some(SourceSelectionCriteria {
replica_modifications: Some(ReplicaModifications {
status: s3s::dto::ReplicaModificationsStatus::from_static(s3s::dto::ReplicaModificationsStatus::DISABLED),
rule.source_selection_criteria = Some(PersistedSourceSelectionCriteria {
replica_modifications: Some(PersistedReplicationStatus {
status: Status::DISABLED.to_string(),
}),
sse_kms_encrypted_objects: None,
});
let replication = ReplicationConfiguration {
let replication = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![rule],
};
let snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
Some(replication.clone()),
@@ -1036,14 +1042,14 @@ mod tests {
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let mut rule = replication_rule();
rule.destination.bucket = arn.to_string();
rule.delete_replication = Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
rule.delete_replication = Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
});
rule.delete_marker_replication = Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)),
rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
status: Some(Status::DISABLED.to_string()),
});
let config = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![rule],
}),
@@ -4794,9 +4794,10 @@ mod tests {
#[tokio::test]
async fn heal_queue_marks_missing_versioning_state_as_missed() {
use super::super::replication_target_boundary::BucketTargets;
use s3s::dto::{
DeleteReplication, DeleteReplicationStatus, Destination, ReplicationConfiguration, ReplicationRule,
ReplicationRuleStatus,
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::{
PersistedReplicationConfiguration, PersistedReplicationDestination, PersistedReplicationRule,
PersistedReplicationStatus,
};
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
@@ -4811,14 +4812,14 @@ mod tests {
..Default::default()
},
ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
rules: vec![PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
delete_replication: Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
}),
destination: Destination {
destination: PersistedReplicationDestination {
bucket: arn.to_string(),
..Default::default()
},
@@ -4828,7 +4829,7 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}],
}),
Some(BucketTargets::default()),
@@ -85,6 +85,8 @@ use http_body_util::StreamBody;
use metrics::counter;
#[cfg(test)]
use rmp_serde;
#[cfg(test)]
use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
use rustfs_s3_types::EventName;
use rustfs_utils::http::{
AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_TAGGING_DIRECTIVE,
@@ -92,8 +94,6 @@ use rustfs_utils::http::{
insert_str, replication_target_versions,
};
use rustfs_utils::{DEFAULT_SIP_HASH_KEY, get_env_usize, sip_hash};
#[cfg(test)]
use s3s::dto::ReplicationConfiguration;
use std::collections::{HashMap, HashSet};
use std::fmt::Display;
use std::pin::Pin;
@@ -6574,9 +6574,10 @@ mod tests {
use super::super::replication_target_boundary::{BucketTarget, BucketTargets};
use super::*;
use s3s::dto::{
BucketVersioningStatus, DeleteReplication, DeleteReplicationStatus, Destination, ExcludedPrefix, ReplicationRule,
ReplicationRuleStatus, VersioningConfiguration,
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use rustfs_gateway_types::persistence::{
PersistedReplicationDestination, PersistedReplicationRule, PersistedReplicationStatus,
};
use std::collections::HashMap;
use time::OffsetDateTime;
@@ -8687,7 +8688,7 @@ mod tests {
..Default::default()
};
let rcfg = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: role.to_string(),
rules: vec![],
}),
@@ -8713,7 +8714,7 @@ mod tests {
ReplicationVersioningStore::install_prefix_state_test_config(
&bucket,
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
);
@@ -8746,22 +8747,20 @@ mod tests {
ReplicationVersioningStore::install_prefix_state_test_config(
&bucket,
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
excluded_prefixes: Some(vec![ExcludedPrefix {
prefix: Some("archive/".to_string()),
}]),
status: Some("Enabled".to_owned()),
excluded_prefixes: Some(vec![Some("archive/".to_string())]),
..Default::default()
},
);
let rcfg = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
rules: vec![PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
delete_replication: Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
}),
destination: Destination {
destination: PersistedReplicationDestination {
bucket: arn.to_string(),
..Default::default()
},
@@ -8771,7 +8770,7 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}],
}),
Some(BucketTargets {
@@ -8807,14 +8806,14 @@ mod tests {
let admitted_arn = "arn:rustfs:replication:us-east-1:target:admitted";
let current_role = "arn:rustfs:replication:us-east-1:target:current";
let rcfg = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: current_role.to_string(),
rules: vec![ReplicationRule {
rules: vec![PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED),
delete_replication: Some(PersistedReplicationStatus {
status: Status::DISABLED.to_string(),
}),
destination: Destination {
destination: PersistedReplicationDestination {
bucket: current_role.to_string(),
..Default::default()
},
@@ -8824,7 +8823,7 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}],
}),
Some(BucketTargets {
@@ -8872,12 +8871,12 @@ mod tests {
#[tokio::test]
async fn heal_rejects_semantically_invalid_replication_config() {
let rcfg = ReplicationConfig::new(
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
rules: vec![PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: None,
destination: Destination {
destination: PersistedReplicationDestination {
bucket: "arn:rustfs:replication:us-east-1:target:bucket".to_string(),
..Default::default()
},
@@ -8887,7 +8886,7 @@ mod tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static("Enabld"),
status: "Enabld".to_owned(),
}],
}),
Some(BucketTargets::default()),
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use s3s::dto::ReplicationConfiguration;
use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
use super::replication_config_boundary::{ObjectOpts, ReplicationConfigurationExt};
use super::replication_filemeta_boundary::ReplicationType;
@@ -20,7 +20,7 @@ use super::replication_filemeta_boundary::ReplicationType;
pub(crate) struct ReplicationTargetConfigBridge;
impl ReplicationTargetConfigBridge {
pub(crate) fn target_is_used_by_rules(config: &ReplicationConfiguration, arn: &str) -> bool {
pub(crate) fn target_is_used_by_rules(config: &PersistedReplicationConfiguration, arn: &str) -> bool {
config
.filter_target_arns(&ObjectOpts {
op_type: ReplicationType::All,
@@ -37,7 +37,7 @@ mod tests {
#[test]
fn target_config_bridge_matches_role_target() {
let config = ReplicationConfiguration {
let config = PersistedReplicationConfiguration {
role: " arn:target ".to_string(),
rules: Vec::new(),
};
@@ -15,7 +15,7 @@
use super::replication_error_boundary::Result;
use crate::bucket::{versioning::VersioningApi as _, versioning_sys::BucketVersioningSys};
#[cfg(test)]
use s3s::dto::VersioningConfiguration;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
#[cfg(test)]
use std::{collections::HashMap, sync::LazyLock, sync::Mutex};
+1 -1
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use s3s::dto::Tag;
use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
use std::cmp::Ordering;
use std::collections::HashMap;
use url::form_urlencoded;
+211 -27
View File
@@ -15,9 +15,23 @@
use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
use crate::error::{Error, Result, StorageError};
use regex::Regex;
use rustfs_gateway_types::cors_tagging::{
CorsTaggingCodecError, PersistedCorsConfiguration, PersistedTagging, parse_cors, parse_tagging, serialize_cors,
serialize_tagging,
};
use rustfs_gateway_types::persistence::{
PersistedAccelerateConfiguration, PersistedBucketEncryptionConfiguration, PersistedBucketLoggingStatus,
PersistedLifecycleConfiguration, PersistedNotificationConfiguration, PersistedObjectLockConfiguration,
PersistedPublicAccessBlockConfiguration, PersistedReplicationConfiguration, PersistedRequestPaymentConfiguration,
PersistedVersioningConfiguration, PersistedWebsiteConfiguration, PersistenceCodecError, parse_accelerate,
parse_bucket_encryption, parse_bucket_logging, parse_lifecycle_legacy_dates, parse_notification, parse_object_lock,
parse_public_access_block, parse_replication, parse_request_payment, parse_versioning, parse_website, serialize_accelerate,
serialize_bucket_encryption, serialize_bucket_logging, serialize_lifecycle_legacy_dates, serialize_notification,
serialize_object_lock, serialize_public_access_block, serialize_replication, serialize_request_payment, serialize_versioning,
serialize_website,
};
use rustfs_utils::WindowsFileNameError;
use rustfs_utils::path::SLASH_SEPARATOR;
use s3s::xml;
use tracing::instrument;
pub fn is_meta_bucketname(name: &str) -> bool {
@@ -74,37 +88,173 @@ pub fn check_valid_bucket_name_strict(bucket_name: &str) -> Result<()> {
check_bucket_name_common(bucket_name, true)
}
// RUSTFS_COMPAT_TODO(s3gate-metadata-xml): the s3s codec reads persisted XML during migration. Remove after every supported writer uses the gateway codec and every retained metadata object and backup archive is verified or rewritten.
pub fn deserialize<T>(input: &[u8]) -> xml::DeResult<T>
where
T: for<'xml> xml::Deserialize<'xml>,
{
let mut d = xml::Deserializer::new(input);
let ans = T::deserialize(&mut d)?;
d.expect_eof()?;
Ok(ans)
/// A bucket configuration persisted as XML in the bucket metadata, together
/// with the codec that reads and writes its stored bytes.
///
/// Every family reads and writes its stored bytes through the gateway
/// persistence codecs (`rustfs_gateway_types::persistence` and
/// `rustfs_gateway_types::cors_tagging`, rustfs/backlog#2745). The codecs keep
/// the persisted bytes the `bucket_config_goldens` pin, so a document the old
/// s3s codec wrote reads back and re-encodes unchanged.
pub trait BucketConfigXml: Sized {
/// Decodes the stored bytes of one configuration document.
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError>;
/// Encodes the value as RustFS stores it.
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError>;
}
#[allow(
dead_code,
reason = "xml serialize helper with no caller in this port; the live sibling is deserialize (backlog#1823)"
)]
pub fn serialize_content<T: xml::SerializeContent>(val: &T) -> xml::SerResult<String> {
let mut buf = Vec::with_capacity(256);
{
let mut ser = xml::Serializer::new(&mut buf);
val.serialize_content(&mut ser)?;
}
Ok(String::from_utf8(buf).unwrap())
/// Why a persisted bucket configuration could not be decoded or encoded.
#[derive(Debug, thiserror::Error)]
pub enum BucketConfigXmlError {
/// A gateway persistence codec refused the bytes or the value.
#[error(transparent)]
Persisted(#[from] PersistenceCodecError),
/// The gateway CORS or Tagging codec refused the bytes or the value.
#[error(transparent)]
CorsTagging(#[from] CorsTaggingCodecError),
}
pub fn serialize<T: xml::Serialize>(val: &T) -> xml::SerResult<Vec<u8>> {
let mut buf = Vec::with_capacity(256);
{
let mut ser = xml::Serializer::new(&mut buf);
val.serialize(&mut ser)?;
/// Decodes one persisted bucket configuration document.
pub fn deserialize<T: BucketConfigXml>(input: &[u8]) -> std::result::Result<T, BucketConfigXmlError> {
T::decode_config_xml(input)
}
/// Encodes one bucket configuration as RustFS persists it.
pub fn serialize<T: BucketConfigXml>(val: &T) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
val.encode_config_xml()
}
impl BucketConfigXml for PersistedReplicationConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_replication(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_replication(self)?)
}
}
impl BucketConfigXml for PersistedLifecycleConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_lifecycle_legacy_dates(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_lifecycle_legacy_dates(self)?)
}
}
// The gateway writers of these families are infallible: a value that exists is
// always encodable, so only the readers can refuse bytes.
impl BucketConfigXml for PersistedAccelerateConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_accelerate(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_accelerate(self))
}
}
impl BucketConfigXml for PersistedBucketLoggingStatus {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_bucket_logging(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_bucket_logging(self))
}
}
impl BucketConfigXml for PersistedCorsConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_cors(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_cors(self))
}
}
impl BucketConfigXml for PersistedNotificationConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_notification(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_notification(self))
}
}
impl BucketConfigXml for PersistedObjectLockConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_object_lock(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_object_lock(self))
}
}
impl BucketConfigXml for PersistedPublicAccessBlockConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_public_access_block(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_public_access_block(self))
}
}
impl BucketConfigXml for PersistedRequestPaymentConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_request_payment(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_request_payment(self))
}
}
impl BucketConfigXml for PersistedBucketEncryptionConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_bucket_encryption(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_bucket_encryption(self))
}
}
impl BucketConfigXml for PersistedTagging {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_tagging(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_tagging(self))
}
}
impl BucketConfigXml for PersistedVersioningConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_versioning(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_versioning(self))
}
}
impl BucketConfigXml for PersistedWebsiteConfiguration {
fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
Ok(parse_website(input)?)
}
fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
Ok(serialize_website(self))
}
Ok(buf)
}
pub fn has_bad_path_component(path: &str) -> bool {
@@ -434,6 +584,40 @@ mod tests {
use super::*;
use proptest::prelude::*;
/// Every bucket-configuration family reads and writes through its own
/// gateway codec (rustfs/backlog#2745): the family's golden document
/// decodes and re-encodes to the same bytes, and a document of another
/// family is refused instead of decoding into an empty configuration.
#[test]
fn every_bucket_config_family_uses_its_own_gateway_codec() {
fn assert_family<T: BucketConfigXml>(family: &str, own: &[u8], foreign: &[u8]) {
let value = deserialize::<T>(own).unwrap_or_else(|err| panic!("{family}: own document must decode: {err}"));
let encoded = serialize(&value).unwrap_or_else(|err| panic!("{family}: value must encode: {err}"));
assert_eq!(encoded, own, "{family}: re-encoding must keep the persisted bytes");
assert!(deserialize::<T>(foreign).is_err(), "{family}: a foreign root element must be refused");
}
macro_rules! golden {
($family:literal) => {
include_bytes!(concat!("../../tests/fixtures/bucket-config-goldens/", $family, "/minio.out.xml"))
};
}
let tagging = golden!("tagging");
let versioning = golden!("versioning");
assert_family::<PersistedAccelerateConfiguration>("accelerate", golden!("accelerate"), tagging);
assert_family::<PersistedBucketLoggingStatus>("logging", golden!("logging"), tagging);
assert_family::<PersistedCorsConfiguration>("cors", golden!("cors"), tagging);
assert_family::<PersistedNotificationConfiguration>("notification", golden!("notification"), tagging);
assert_family::<PersistedObjectLockConfiguration>("object_lock", golden!("object_lock"), tagging);
assert_family::<PersistedPublicAccessBlockConfiguration>("public_access_block", golden!("public_access_block"), tagging);
assert_family::<PersistedRequestPaymentConfiguration>("request_payment", golden!("request_payment"), tagging);
assert_family::<PersistedBucketEncryptionConfiguration>("encryption", golden!("encryption"), tagging);
assert_family::<PersistedTagging>("tagging", tagging, versioning);
assert_family::<PersistedVersioningConfiguration>("versioning", versioning, tagging);
assert_family::<PersistedWebsiteConfiguration>("website", golden!("website"), tagging);
}
/// rustfs#5785: keys whose path segments exceed the on-disk NAME_MAX
/// budget must be rejected up front as ObjectNameInvalid (4xx), not leak
/// ENAMETOOLONG as InternalError 500 from the disk layer.
+17 -25
View File
@@ -12,8 +12,8 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use rustfs_utils::string::match_simple;
use s3s::dto::{BucketVersioningStatus, VersioningConfiguration};
pub trait VersioningApi {
fn enabled(&self) -> bool;
@@ -28,10 +28,10 @@ pub trait VersioningApi {
impl VersioningApi for VersioningConfiguration {
fn enabled(&self) -> bool {
self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED))
self.status.as_deref() == Some("Enabled")
}
fn prefix_enabled(&self, prefix: &str) -> bool {
if self.status != Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)) {
if self.status.as_deref() != Some("Enabled") {
return false;
}
@@ -44,12 +44,10 @@ impl VersioningApi for VersioningConfiguration {
}
if let Some(ref excluded_prefixes) = self.excluded_prefixes {
for p in excluded_prefixes.iter() {
if let Some(ref sprefix) = p.prefix {
let pattern = format!("{sprefix}*");
if match_simple(&pattern, prefix) {
return false;
}
for sprefix in excluded_prefixes.iter().flatten() {
let pattern = format!("{sprefix}*");
if match_simple(&pattern, prefix) {
return false;
}
}
}
@@ -58,11 +56,11 @@ impl VersioningApi for VersioningConfiguration {
}
fn prefix_suspended(&self, prefix: &str) -> bool {
if self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED)) {
if self.status.as_deref() == Some("Suspended") {
return true;
}
if self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)) {
if self.status.as_deref() == Some("Enabled") {
if prefix.is_empty() {
return false;
}
@@ -75,12 +73,10 @@ impl VersioningApi for VersioningConfiguration {
}
if let Some(ref excluded_prefixes) = self.excluded_prefixes {
for p in excluded_prefixes.iter() {
if let Some(ref sprefix) = p.prefix {
let pattern = format!("{sprefix}*");
if match_simple(&pattern, prefix) {
return true;
}
for sprefix in excluded_prefixes.iter().flatten() {
let pattern = format!("{sprefix}*");
if match_simple(&pattern, prefix) {
return true;
}
}
}
@@ -92,14 +88,12 @@ impl VersioningApi for VersioningConfiguration {
self.prefix_enabled(prefix) || self.prefix_suspended(prefix)
}
fn suspended(&self) -> bool {
self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED))
self.status.as_deref() == Some("Suspended")
}
}
#[cfg(test)]
mod tests {
use s3s::dto::{BucketVersioningStatus, ExcludedPrefix};
use super::*;
struct LegacyVersioning;
@@ -134,10 +128,8 @@ mod tests {
#[test]
fn delete_state_treats_excluded_prefixes_as_unversioned() {
let config = VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
excluded_prefixes: Some(vec![ExcludedPrefix {
prefix: Some("archive/".to_string()),
}]),
status: Some("Enabled".to_owned()),
excluded_prefixes: Some(vec![Some("archive/".to_string())]),
..Default::default()
};
@@ -146,7 +138,7 @@ mod tests {
assert_eq!(config.delete_state("live/object"), (true, false));
let suspended = VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED)),
status: Some("Suspended".to_owned()),
..Default::default()
};
assert_eq!(suspended.delete_state("archive/object"), (false, true));
+2 -2
View File
@@ -17,7 +17,7 @@ use super::metadata::unreadable_config_refusal;
use super::{metadata_sys::get_bucket_metadata_sys, versioning::VersioningApi};
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::Result;
use s3s::dto::VersioningConfiguration;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use tracing::{error, warn};
pub struct BucketVersioningSys {}
@@ -163,7 +163,7 @@ mod tests {
fn enabled() -> VersioningConfiguration {
VersioningConfiguration {
status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
}
}
@@ -747,16 +747,6 @@ async fn list_path_raw_inner(
// Start every missing head read in the same round so one stalled
// disk cannot multiply the wait budget by the erasure-set width.
// Outcomes are still consumed below in stable disk-index order.
if !revjob_rx.is_cancelled() {
for (i, reader) in readers.iter_mut().enumerate() {
if errs[i].is_none()
&& pending_entries[i].is_none()
&& let Some(Ok(Some(entry))) = reader.try_peek_buffered()
{
pending_entries[i] = Some(entry);
}
}
}
let concurrent_peeks = readers.iter_mut().enumerate().filter_map(|(i, reader)| {
if errs[i].is_some() || pending_entries[i].is_some() {
return None;
@@ -888,7 +878,7 @@ async fn list_path_raw_inner(
}
if entry.name == current.name {
// If exact match, we agree.
if current.matches_strict_borrowed(&entry) {
if current.matches_strict(&entry) {
top_entries[i] = Some(entry);
agree += 1;
} else {
@@ -996,9 +986,7 @@ async fn list_path_raw_inner(
if agree == readers.len() && !opts.preserve_replica_metadata {
for r in readers.iter_mut() {
if !r.skip_current() {
let _ = r.skip(1).await;
}
let _ = r.skip(1).await;
}
if let Some(agreed_fn) = opts.agreed.as_ref() {
@@ -1021,7 +1009,7 @@ async fn list_path_raw_inner(
}
for (i, r) in readers.iter_mut().enumerate() {
if top_entries[i].is_some() && !r.skip_current() {
if top_entries[i].is_some() {
let _ = r.skip(1).await;
}
}
@@ -1965,6 +1965,15 @@ impl PeerRestClient {
Ok((self.topology_member.clone(), epoch))
}
pub async fn probe_restore_copyback_capability(&self, topology_fingerprint: String) -> Result<(String, Uuid)> {
let probe = rustfs_protos::restore_copyback_capability_probe(Uuid::new_v4().as_bytes());
let result = self
.heal_control(rustfs_protos::HEAL_CONTROL_PROTOCOL_VERSION, topology_fingerprint, probe)
.await?;
let epoch = decode_remote_version_state_capability(&self.topology_member, &result)?;
Ok((self.topology_member.clone(), epoch))
}
pub async fn probe_cross_pool_fence(&self, topology_fingerprint: String) -> Result<(String, u32, Uuid)> {
let mut probe = rustfs_protos::CROSS_POOL_FENCE_CAPABILITY_PROBE_PREFIX.to_vec();
probe.extend_from_slice(Uuid::new_v4().as_bytes());
-1
View File
@@ -3506,7 +3506,6 @@ mod tests {
replication_decision: String::new(),
checksum: None,
multipart_completion_replayed: false,
backfilled_old_current_size: None,
}
}
}
+37 -167
View File
@@ -84,12 +84,13 @@ use rand::RngExt as _;
use rmp_serde::Deserializer;
use rmp_serde::Serializer;
use rustfs_filemeta::{FileInfo, FileInfoVersions, MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams};
use rustfs_gateway_types::persistence::PersistedObjectLockConfiguration as ObjectLockConfiguration;
use rustfs_gateway_types::persistence::{PersistedLifecycleConfiguration, PersistedReplicationConfiguration};
use rustfs_heal_contracts::heal_channel::HealOpts;
use rustfs_utils::crypto::{hex_sha256, is_sha256_checksum};
use rustfs_utils::path::{
decode_dir_object, encode_dir_object, path_join, path_to_bucket_object, path_to_bucket_object_with_base_path,
};
use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration, ReplicationConfiguration};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::collections::{HashMap, HashSet};
@@ -3720,45 +3721,33 @@ fn is_decommission_copy_quorum_deferral(err: &Error) -> bool {
data_movement::data_movement_stage_source_as::<DecommissionCopyQuorumFailure>(err).is_some()
}
/// A copy whose object namespace lock remained unavailable through its
/// in-place retries, retained as a typed deferral.
/// The storage error of a version copy that kept timing out on a namespace
/// lock through its in-place retries, kept typed so the entry failure policy
/// defers it.
#[derive(Debug, thiserror::Error)]
#[error("{0}")]
struct DecommissionCopyLockUnavailable(#[source] Error);
struct DecommissionCopyLockTimeout(#[source] Error);
fn is_decommission_copy_lock_unavailable_error(err: &Error, bucket: &str, object: &str) -> bool {
let source = data_movement::data_movement_stage_source(err).unwrap_or(err);
match source {
Error::Lock(rustfs_lock::LockError::Timeout { .. }) => true,
Error::NamespaceLockQuorumUnavailable {
mode: "read",
bucket: lock_bucket,
object: lock_object,
..
} => {
data_movement::data_movement_error_is_stage(err, "prepare_put_object")
&& lock_bucket == bucket
&& lock_object == object
}
_ => false,
}
fn is_decommission_copy_lock_timeout_error(err: &Error) -> bool {
let err = data_movement::data_movement_stage_source(err).unwrap_or(err);
matches!(err, Error::Lock(rustfs_lock::LockError::Timeout { .. }))
}
/// A version copy that could not acquire its namespace lock leaves the source
/// A version copy that timed out acquiring a namespace lock leaves the source
/// in place; the next attempt copies the version again or finds an equivalent
/// committed target. The lock is held by another mutation of the key or, when
/// this node restarted mid-copy, by its previous incarnation until that lock's
/// lease expires on the peers. Either clears by waiting, so the entry is
/// deferred instead of being counted as a permanent per-object failure.
fn decommission_copy_lock_unavailable_deferral(stage: &str, bucket: &str, object: &str, err: Error) -> Error {
fn decommission_copy_lock_timeout_deferral(stage: &str, bucket: &str, object: &str, err: Error) -> Error {
data_movement::data_movement_context_error(
format!("decommission {stage} could not acquire a namespace lock for {bucket}/{object}: {err}"),
DecommissionCopyLockUnavailable(err),
format!("decommission {stage} timed out on a namespace lock for {bucket}/{object}: {err}"),
DecommissionCopyLockTimeout(err),
)
}
fn is_decommission_copy_lock_unavailable_deferral(err: &Error) -> bool {
data_movement::data_movement_stage_source_as::<DecommissionCopyLockUnavailable>(err).is_some()
fn is_decommission_copy_lock_timeout_deferral(err: &Error) -> bool {
data_movement::data_movement_stage_source_as::<DecommissionCopyLockTimeout>(err).is_some()
}
/// The deferral of a version copy whose in-place retries ended on `error`, if
@@ -3768,8 +3757,8 @@ fn take_decommission_copy_deferral(stage: &str, bucket: &str, object: &str, erro
return Some(decommission_copy_quorum_deferral(stage, bucket, object, err));
}
error
.take_if(|err| is_decommission_copy_lock_unavailable_error(err, bucket, object))
.map(|err| decommission_copy_lock_unavailable_deferral(stage, bucket, object, err))
.take_if(|err| is_decommission_copy_lock_timeout_error(err))
.map(|err| decommission_copy_lock_timeout_deferral(stage, bucket, object, err))
}
fn resolve_decommission_entry_reload_result(result: Result<()>, bucket: &str, object_name: &str) -> Result<()> {
@@ -5834,13 +5823,13 @@ enum DecommissionEntryOutcome {
/// entry is done and must not be counted as a failure or escalated. A quorum
/// failure of the source cleanup is deferred too: the copied source is still
/// in place and the next attempt cleans it up again. So is a version copy that
/// lost drive quorum or could not acquire an object namespace lock through its in-place
/// lost drive quorum or timed out on a namespace lock through its in-place
/// retries: the source still holds the version and the next attempt copies it
/// again. Every other failure stays terminal.
fn resolve_decommission_entry_failure(err: Error) -> Result<DecommissionEntryOutcome> {
if is_decommission_source_cleanup_retryable_error(&err)
|| is_decommission_copy_quorum_deferral(&err)
|| is_decommission_copy_lock_unavailable_deferral(&err)
|| is_decommission_copy_lock_timeout_deferral(&err)
{
return Ok(DecommissionEntryOutcome::Deferred(err));
}
@@ -12895,7 +12884,7 @@ pub(crate) async fn should_skip_lifecycle_for_data_movement(
store: Arc<ECStore>,
bucket: &str,
version: &rustfs_filemeta::FileInfo,
lifecycle_config: Option<&BucketLifecycleConfiguration>,
lifecycle_config: Option<&PersistedLifecycleConfiguration>,
object_lock_config: Option<&ObjectLockConfiguration>,
apply_actions: bool,
event_source: &LcEventSrc,
@@ -16824,9 +16813,9 @@ impl ECStore {
generation: OffsetDateTime,
bucket: String,
set: Arc<SetDisks>,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
object_lock_config: Option<ObjectLockConfiguration>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
source_changed_exhaustions: Arc<AtomicUsize>,
entry_budget: Arc<Semaphore>,
@@ -16998,9 +16987,9 @@ impl ECStore {
mut deferred_entry: DeferredDecommissionEntry,
bucket: String,
set: Arc<SetDisks>,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
object_lock_config: Option<ObjectLockConfiguration>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
source_changed_exhaustions: Arc<AtomicUsize>,
entry_budget: Arc<Semaphore>,
@@ -17107,9 +17096,9 @@ impl ECStore {
generation: OffsetDateTime,
set: Arc<SetDisks>,
bi: DecomBucketInfo,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
object_lock_config: Option<ObjectLockConfiguration>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
source_changed_exhaustions: Arc<AtomicUsize>,
entry_budget: Arc<Semaphore>,
@@ -17784,9 +17773,9 @@ impl ECStore {
entry: MetaCacheEntry,
bucket: String,
set: Arc<SetDisks>,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
object_lock_config: Option<ObjectLockConfiguration>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
source_changed_exhaustions: Arc<AtomicUsize>,
attempts: &mut usize,
@@ -17970,9 +17959,9 @@ impl ECStore {
entry: MetaCacheEntry,
bucket: String,
set: Arc<SetDisks>,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
object_lock_config: Option<ObjectLockConfiguration>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
entry_attempt: usize,
scanner_backlog_handoff_attempt: usize,
@@ -19258,7 +19247,7 @@ impl ECStore {
entry: MetaCacheEntry,
bucket: String,
set: Arc<SetDisks>,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
) -> Result<()> {
let expected_bucket_incarnation_id = if is_meta_bucketname(&bucket) {
None
@@ -28365,7 +28354,7 @@ mod tests {
"one version must commit before the next version times out"
);
assert!(
matches!(&outcome, DecommissionEntryOutcome::Deferred(err) if is_decommission_copy_lock_unavailable_deferral(err)),
matches!(&outcome, DecommissionEntryOutcome::Deferred(err) if is_decommission_copy_lock_timeout_deferral(err)),
"{outcome:?}"
);
assert_eq!(
@@ -28399,88 +28388,6 @@ mod tests {
assert_eq!(actual, b"written after versioning was enabled");
}
/// A target lookup that loses its read-lock quorum must retain the source
/// and resume an entry after an earlier version has already committed.
#[tokio::test]
#[serial_test::serial]
async fn decommission_defers_target_lookup_read_lock_quorum_and_resumes() {
use tokio::io::AsyncReadExt as _;
let fixture = NullVersionDecommission::new(NullVersionTop::NewerVersion).await;
let object = fixture.object;
let attempted = Arc::new(AtomicUsize::new(0));
let fault = DecommissionTestFaultGuard::install_with_error(
Arc::new({
let attempted = Arc::clone(&attempted);
move |stage, _, faulted, _, success| {
stage == "before_target_commit"
&& faulted == object
&& success
&& attempted.fetch_add(1, Ordering::SeqCst) > 0
}
}),
Some(Arc::new(|_, bucket, object| {
data_movement::data_movement_stage_error_for_test(
"decommission_object",
"prepare_put_object",
bucket,
object,
Error::NamespaceLockQuorumUnavailable {
mode: "read",
bucket: bucket.to_owned(),
object: object.to_owned(),
required: 1,
achieved: 0,
},
)
})),
);
let (outcome, _) = fixture.run_entry().await;
assert!(
matches!(&outcome, DecommissionEntryOutcome::Deferred(err) if is_decommission_copy_lock_unavailable_deferral(err)),
"{outcome:?}"
);
fixture.assert_source_retained().await;
assert_eq!(
fixture
.target_versions()
.await
.expect("first version committed")
.versions
.len(),
1
);
assert_eq!(
fixture.store.pool_meta.read().await.pools[0]
.decommission
.as_ref()
.expect("active decommission")
.items_decommission_failed,
0
);
drop(fault);
let (outcome, _) = fixture.run_entry().await;
assert!(matches!(outcome, DecommissionEntryOutcome::Complete), "{outcome:?}");
fixture.assert_moved().await;
let mut reader = fixture.store.pools[1]
.get_object_reader(
fixture.bucket,
fixture.object,
None,
HeaderMap::new(),
&ObjectOptions {
version_id: Some(fixture.seeded.versions[0].version_id.expect("newer version").to_string()),
..Default::default()
},
)
.await
.expect("the newer version must remain readable");
let mut actual = Vec::new();
reader.stream.read_to_end(&mut actual).await.expect("read the newer version");
assert_eq!(actual, b"written after versioning was enabled");
}
/// A namespace lock still held on the target key, as by a copy whose node
/// was killed mid-write, makes target selection fail with a typed
/// object-lock timeout. A real held lock, not an injected error, must defer
@@ -28523,7 +28430,7 @@ mod tests {
let (outcome, _) = fixture.run_entry().await;
assert!(
matches!(&outcome, DecommissionEntryOutcome::Deferred(err) if is_decommission_copy_lock_unavailable_deferral(err)),
matches!(&outcome, DecommissionEntryOutcome::Deferred(err) if is_decommission_copy_lock_timeout_deferral(err)),
"{outcome:?}"
);
assert_eq!(
@@ -28932,17 +28839,17 @@ mod tests {
timeout(),
);
for err in [timeout(), prepare] {
assert!(is_decommission_copy_lock_unavailable_error(&err, "bucket-a", "object-a"), "{err}");
assert!(is_decommission_copy_lock_timeout_error(&err), "{err}");
assert!(!is_decommission_copy_quorum_error(&err), "{err}");
let mut error = Some(err);
let deferred = take_decommission_copy_deferral("migrate_object", "bucket-a", "object-a", &mut error)
.expect("a lock timeout copy must be deferred");
assert!(error.is_none());
assert!(is_decommission_copy_lock_unavailable_deferral(&deferred));
assert!(is_decommission_copy_lock_timeout_deferral(&deferred));
assert!(!is_decommission_copy_quorum_deferral(&deferred));
let failure = data_movement::data_movement_stage_source_as::<DecommissionCopyLockUnavailable>(&deferred)
let failure = data_movement::data_movement_stage_source_as::<DecommissionCopyLockTimeout>(&deferred)
.expect("the deferred error must keep its lock-timeout origin");
assert!(is_decommission_copy_lock_unavailable_error(&failure.0, "bucket-a", "object-a"));
assert!(is_decommission_copy_lock_timeout_error(&failure.0));
assert!(matches!(
resolve_decommission_entry_failure(deferred),
Ok(DecommissionEntryOutcome::Deferred(_))
@@ -28967,43 +28874,6 @@ mod tests {
}
}
#[test]
fn decommission_copy_defers_only_the_matching_object_read_lock_quorum() {
let quorum = |mode, bucket: &str, object: &str| Error::NamespaceLockQuorumUnavailable {
mode,
bucket: bucket.to_owned(),
object: object.to_owned(),
required: 1,
achieved: 0,
};
let wrap = |stage, err| {
data_movement::data_movement_stage_error_for_test("decommission_object", stage, "bucket-a", "object-a", err)
};
let mut error = Some(wrap("prepare_put_object", quorum("read", "bucket-a", "object-a")));
let deferred = take_decommission_copy_deferral("migrate_object", "bucket-a", "object-a", &mut error)
.expect("matching read-lock quorum must defer");
assert!(error.is_none());
assert!(matches!(
resolve_decommission_entry_failure(deferred),
Ok(DecommissionEntryOutcome::Deferred(_))
));
for err in [
quorum("read", "bucket-a", "object-a"),
wrap("source_recheck", quorum("read", "bucket-a", "object-a")),
wrap("prepare_put_object", quorum("write", "bucket-a", "object-a")),
wrap(
"prepare_put_object",
quorum("exact_delete_capacity_reconciliation", "bucket-a", "object-a"),
),
wrap("prepare_put_object", quorum("read", "other-bucket", "object-a")),
wrap("prepare_put_object", quorum("read", "bucket-a", "other-object")),
] {
let mut error = Some(err);
assert!(take_decommission_copy_deferral("migrate_object", "bucket-a", "object-a", &mut error).is_none());
assert!(error.is_some(), "unrelated lock failures must stay terminal");
}
}
#[test]
fn decommission_target_capacity_error_accepts_wrapped_capacity_errors() {
let disk_full = Error::other(format!("decommission_object: put_object failed for bucket/object: {}", Error::DiskFull));
+40
View File
@@ -65,6 +65,38 @@ use uuid::Uuid;
const LIST_MULTIPART_SETS_CONCURRENCY: usize = 4;
#[cfg(test)]
type TestSetLockers = HashMap<usize, Vec<Arc<dyn rustfs_lock::LockClient>>>;
#[cfg(test)]
static TEST_SET_LOCKERS: std::sync::LazyLock<std::sync::Mutex<TestSetLockers>> =
std::sync::LazyLock::new(|| std::sync::Mutex::new(HashMap::new()));
/// Keep a fixture's real locker domain scoped to its owning instance.
#[cfg(test)]
pub(crate) struct ScopedSetLockers(Arc<InstanceContext>);
#[cfg(test)]
impl ScopedSetLockers {
pub(crate) fn install(ctx: Arc<InstanceContext>, lockers: Vec<Arc<dyn rustfs_lock::LockClient>>) -> Self {
let key = Arc::as_ptr(&ctx) as usize;
let mut domains = TEST_SET_LOCKERS.lock().expect("test set lockers mutex");
assert!(!domains.contains_key(&key), "the instance already owns a test locker domain");
domains.insert(key, lockers);
Self(ctx)
}
}
#[cfg(test)]
impl Drop for ScopedSetLockers {
fn drop(&mut self) {
TEST_SET_LOCKERS
.lock()
.expect("test set lockers mutex")
.remove(&(Arc::as_ptr(&self.0) as usize));
}
}
fn is_idempotent_delete_prefix_error(err: &Error) -> bool {
is_err_object_not_found(err) || is_err_strict_volume_not_found(err)
}
@@ -266,6 +298,14 @@ impl Sets {
}
}
#[cfg(test)]
let pool_lockers = TEST_SET_LOCKERS
.lock()
.expect("test set lockers mutex")
.get(&(Arc::as_ptr(&instance_ctx) as usize))
.cloned()
.unwrap_or_else(|| pool_lockers.clone());
let set_disks = SetDisks::new_with_instance_ctx(
runtime_sources::local_node_name().await,
Arc::new(RwLock::new(set_drive)),
+1 -17
View File
@@ -605,7 +605,6 @@ fn resolve_data_movement_abort_result(
#[derive(Debug)]
struct DataMovementStageError {
rendered: String,
stage: Option<String>,
source: Box<dyn std::error::Error + Send + Sync>,
}
@@ -627,7 +626,6 @@ where
{
Error::other(DataMovementStageError {
rendered,
stage: None,
source: Box::new(err),
})
}
@@ -636,21 +634,7 @@ fn data_movement_stage_error<E>(op_label: &str, stage: &str, bucket: &str, objec
where
E: std::error::Error + Send + Sync + 'static,
{
Error::other(DataMovementStageError {
rendered: format!("{op_label}: {stage} failed for {bucket}/{object}: {err}"),
stage: Some(stage.to_owned()),
source: Box::new(err),
})
}
pub(crate) fn data_movement_error_is_stage(err: &Error, stage: &str) -> bool {
let Error::Io(io_err) = err else {
return false;
};
io_err
.get_ref()
.and_then(|source| source.downcast_ref::<DataMovementStageError>())
.is_some_and(|source| source.stage.as_deref() == Some(stage))
data_movement_context_error(format!("{op_label}: {stage} failed for {bucket}/{object}: {err}"), err)
}
#[cfg(test)]
+9 -4
View File
@@ -2398,10 +2398,6 @@ impl LocalDiskWrapper {
.await
}
pub(crate) fn has_clean_active_multipart_source(&self, upload_or_part_path: &str, part_path: bool) -> bool {
self.disk.has_clean_active_multipart_source(upload_or_part_path, part_path)
}
pub(crate) async fn try_publish_prepared_delete_local(
&self,
volume: &str,
@@ -2920,6 +2916,15 @@ impl DiskAPI for LocalDiskWrapper {
})
}
async fn bucket_creation_witness(&self, volume: &str) -> Result<Option<super::os::NativeBucketCreationWitness>> {
self.track_disk_health_with_op(
"bucket_creation_witness",
|| async { self.disk.bucket_creation_witness(volume).await },
Duration::ZERO,
)
.await
}
async fn make_volume(&self, volume: &str) -> Result<()> {
// Scoped heal must drain directory creation before releasing its lifecycle owner.
let timeout = if crate::store::bucket_heal_scope(volume).is_some() {
+7
View File
@@ -92,6 +92,9 @@ pub const O_APPEND: FileMode = 0x00400;
// create_new: bool,
pub async fn open_file(path: impl AsRef<Path>, mode: FileMode) -> io::Result<File> {
if mode & O_CREATE != 0 {
super::os::record_native_publication(path.as_ref());
}
let path = path.as_ref().to_path_buf();
native_io(move || {
let mut options = std::fs::OpenOptions::new();
@@ -144,6 +147,7 @@ pub fn create_dir_all_std(path: &Path) -> io::Result<()> {
}
pub async fn make_dir_all(path: impl AsRef<Path>) -> io::Result<()> {
super::os::record_native_publication(path.as_ref());
let path = path.as_ref().to_path_buf();
native_io(move || create_dir_all_std(&path)).await
}
@@ -205,11 +209,13 @@ pub fn remove_all_std(path: impl AsRef<Path>) -> io::Result<()> {
#[cfg(test)]
pub async fn mkdir(path: impl AsRef<Path>) -> io::Result<()> {
super::os::record_native_publication(path.as_ref());
let path = path.as_ref().to_path_buf();
native_io(move || std::fs::create_dir(path)).await
}
pub async fn rename(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()> {
super::os::record_native_publication(to.as_ref());
let from = from.as_ref().to_path_buf();
let to = to.as_ref().to_path_buf();
native_io(move || std::fs::rename(from, to)).await
@@ -217,6 +223,7 @@ pub async fn rename(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<
#[cfg(any(not(windows), test))]
pub fn rename_std(from: impl AsRef<Path>, to: impl AsRef<Path>) -> io::Result<()> {
super::os::record_native_publication(to.as_ref());
std::fs::rename(from, to)
}
File diff suppressed because it is too large Load Diff
+125 -553
View File
@@ -175,30 +175,17 @@ pub(super) fn create_local_inline_rollback_backup(
dst_file_path: &Path,
staging_file_path: &Path,
old_metadata: &[u8],
sync: bool,
) -> std::io::Result<PathBuf> {
let Some(staging_parent) = staging_file_path.parent() else {
return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing staging metadata parent"));
};
let backup_path = staging_parent.join(STORAGE_FORMAT_FILE_BACKUP);
remove_file_if_exists(&backup_path)?;
if should_fail_local_inline_rollback_hardlink(dst_file_path) || std::fs::hard_link(dst_file_path, &backup_path).is_err() {
let write_res = (|| -> std::io::Result<()> {
#[cfg(test)]
run_owned_file_before_open(&backup_path);
let mut file = std::fs::File::create(&backup_path)?;
std::io::Write::write_all(&mut file, old_metadata)?;
if sync && !old_metadata.is_empty() {
file.sync_data()?;
}
Ok(())
})();
if let Err(err) = write_res {
let _ = remove_file_if_exists(&backup_path);
return Err(err);
}
} else if sync && !old_metadata.is_empty() {
std::fs::File::open(&backup_path)?.sync_data()?;
if (should_fail_local_inline_rollback_hardlink(dst_file_path) || std::fs::hard_link(dst_file_path, &backup_path).is_err())
&& let Err(err) = std::fs::write(&backup_path, old_metadata)
{
let _ = remove_file_if_exists(&backup_path);
return Err(err);
}
Ok(backup_path)
}
@@ -369,12 +356,9 @@ fn prepare_rename_commit_and_destination_metadata_std(
#[cfg(target_os = "linux")]
enum FusedCommitAttempt {
Committed(Box<RenameDataResp>),
Committed(RenameDataResp),
CommitErr(DiskError),
RestoreDataSource(os::RenameCommitGuard, DiskError),
Fallback(os::RenameCommitGuard, Option<Bytes>, Box<FileInfo>),
#[cfg(test)]
ResumePanic(Box<dyn std::any::Any + Send + 'static>),
Fallback(os::RenameCommitGuard, Option<Bytes>, FileInfo),
}
async fn prepare_rename_commit_and_destination_metadata(
@@ -556,9 +540,7 @@ impl LocalDisk {
None
};
let multipart_publication = if let Some(staging) = multipart_staging.as_ref() {
Some(Arc::new(
pin_multipart_future(|| self.prepare_native_multipart_publication_deferred(src_path, &mut fi, staging)).await?,
))
Some(pin_multipart_future(|| self.prepare_native_multipart_publication(src_path, &mut fi, staging)).await?)
} else {
None
};
@@ -695,8 +677,20 @@ impl LocalDisk {
};
let dst_file_path = self.io_get_object_open_path(dst_volume, format!("{}/{}", dst_path, STORAGE_FORMAT_FILE).as_str())?;
// Multipart restore completion has validated staged local parts,
// while its public status stays pending until finalization.
let pending_multipart_restore = src_volume == crate::disk::RUSTFS_META_MULTIPART_BUCKET
&& fi
.metadata
.get(rustfs_filemeta::metadata_keys::RESTORE)
.and_then(|value| rustfs_filemeta::parse_restore_obj_status(value).ok())
.is_some_and(|status| status.is_restore_in_progress == Some(true));
if pending_multipart_restore && fi.data_dir.is_none_or(|dir| dir.is_nil()) {
return Err(DiskError::FileCorrupt);
}
// data_dir path (only used by the non-inline commit branch)
let has_data_dir_path = if no_inline && !fi.is_remote() {
let has_data_dir_path = if no_inline && (!fi.is_remote() || pending_multipart_restore) {
if let Some(data_dir) = fi
.data_dir
.map(|dir| rustfs_utils::path::retain_slash(dir.to_string().as_str()))
@@ -759,6 +753,11 @@ impl LocalDisk {
&& !fi_healing
&& !os::dst_dir_fsync_group_commit_enabled()
&& !os::file_fdatasync_group_commit_enabled()
&& !super::has_rename_data_test_failpoints(dst_path)
&& !crash_inject::is_armed(CrashPoint::RenameAfterDataRename, dst_path)
&& !crash_inject::is_armed(CrashPoint::RenameAfterBackupBeforeMetaCommit, dst_path)
&& !crash_inject::is_armed(CrashPoint::RenameAfterMetaCommit, dst_path)
&& !super::has_delete_version_test_failpoints()
{
#[cfg(test)]
if !no_inline {
@@ -798,11 +797,10 @@ impl LocalDisk {
&dst_volume_dir_owned,
&publication_root_owned,
)?;
// An existing xl.meta that replaces an existing version slot
// requires a rollback directory; appending a new version keeps
// an inline rollback backup for post-rename directory sync errors.
if is_cancelled() {
return Ok(FusedCommitAttempt::Fallback(guard, has_dst_buf, Box::new(fi_moved)));
// An existing xl.meta can share versions or require rollback
// restoration when a post-rename directory sync fails.
if is_cancelled() || (sync_commit_metadata && has_dst_buf.is_some()) {
return Ok(FusedCommitAttempt::Fallback(guard, has_dst_buf, fi_moved));
}
let mut xlmeta = FileMeta::new();
let mut dst_meta_unparsable = false;
@@ -825,19 +823,7 @@ impl LocalDisk {
}
});
if rollback_data_dir.is_some() {
return Ok(FusedCommitAttempt::Fallback(guard, has_dst_buf, Box::new(fi_moved)));
}
#[cfg(test)]
let probe_active = os::file_sync_probe::is_active();
#[cfg(not(test))]
let probe_active = false;
if no_inline
&& sync_data_shards
&& let Some((src_data_path, _)) = has_data_dir_path_owned.as_ref()
&& (probe_active || !super::is_presynced_tmp_data_dir(src_data_path))
&& os::regular_files(src_data_path).is_ok_and(|files| files.len() >= os::PARALLEL_FILE_SYNC_THRESHOLD)
{
return Ok(FusedCommitAttempt::Fallback(guard, has_dst_buf, Box::new(fi_moved)));
return Ok(FusedCommitAttempt::Fallback(guard, has_dst_buf, fi_moved));
}
let (old_current_size, old_current_source_checked, old_current_source) = observe_old_current(
has_dst_buf.is_some(),
@@ -856,31 +842,13 @@ impl LocalDisk {
skip_parent = dst_parent_owned.clone();
}
#[cfg(test)]
run_owned_file_before_open(&src_file_path_owned);
let mut prepared_metadata_source =
os::create_prepared_rename_source_with_commit_guard(&src_file_path_owned, &dst_file_path_owned, &guard)?;
prepared_metadata_source.write_all(&new_dst_buf, false)?;
if !no_inline {
run_inline_preparation_before_backup(&dst_path_str);
}
let mut staged_rollback_path = None;
if let Some(ref old_metadata) = has_dst_buf
&& (sync_commit_metadata || cfg!(test))
{
let backup_path = create_local_inline_rollback_backup(
&dst_file_path_owned,
&src_file_path_owned,
old_metadata,
sync_commit_metadata,
)?;
staged_rollback_path = Some(StagedRollbackBackup(backup_path));
}
prepared_metadata_source.write_all(&new_dst_buf, sync_commit_metadata)?;
if no_inline {
if sync_data_shards && let Some((src_data_path, _)) = has_data_dir_path_owned.as_ref() {
let sync_result = if !probe_active && super::take_presynced_tmp_data_dir(src_data_path) {
let sync_result = if super::take_presynced_tmp_data_dir(src_data_path) {
// The payload was synced, but its part.1 directory entry
// must also be persisted before moving the data directory.
os::fsync_dir_std(src_data_path)
@@ -890,157 +858,49 @@ impl LocalDisk {
if let Err(err) = sync_result
&& err.kind() != ErrorKind::NotFound
{
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
return Err(err);
}
}
if sync_commit_metadata && let Err(err) = prepared_metadata_source.sync_data() {
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
return Err(err);
}
} else if sync_commit_metadata && let Err(err) = prepared_metadata_source.sync_data() {
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
return Err(err);
}
if is_cancelled() {
drop(prepared_metadata_source);
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
return Err(std::io::Error::new(ErrorKind::Interrupted, "commit cancelled"));
}
let guard = match super::remove_dst_base_before_commit_std(
&dst_path_str,
guard,
&src_parent_owned,
&dst_parent_owned,
&dst_volume_dir_owned,
&publication_root_owned,
) {
Ok(guard) => guard,
Err(err) => {
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
return Ok(FusedCommitAttempt::CommitErr(err));
}
};
if should_remove_staged_meta_before_commit(&dst_path_str) {
drop(prepared_metadata_source);
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
std::fs::remove_file(&src_file_path_owned)?;
return Ok(FusedCommitAttempt::CommitErr(DiskError::FileNotFound));
}
if no_inline {
if let Some((src_data_path, dst_data_path)) = has_data_dir_path_owned.as_ref()
&& let Err(err) = os::rename_all_with_commit_guard_on_thread(src_data_path, dst_data_path, &guard)
{
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
return Ok(FusedCommitAttempt::RestoreDataSource(guard, err));
return Ok(FusedCommitAttempt::CommitErr(err));
}
#[cfg(test)]
if has_data_dir_path_owned.is_some()
&& let Err(panic_payload) = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
run_rename_data_after_first_publication(&root_owned, &dst_volume_str, &dst_path_str);
}))
{
return Ok(FusedCommitAttempt::ResumePanic(panic_payload));
}
if crash_inject::should_crash_at(CrashPoint::RenameAfterDataRename, &dst_path_str) {
return Ok(FusedCommitAttempt::CommitErr(DiskError::Unexpected));
}
if should_fail_before_old_metadata_backup(&dst_path_str) {
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
return Ok(FusedCommitAttempt::RestoreDataSource(guard, DiskError::Unexpected));
}
if crash_inject::should_crash_at(CrashPoint::RenameAfterBackupBeforeMetaCommit, &dst_path_str) {
return Ok(FusedCommitAttempt::CommitErr(DiskError::Unexpected));
if has_data_dir_path_owned.is_some() {
run_rename_data_after_first_publication(&root_owned, &dst_volume_str, &dst_path_str);
}
}
let rename_res = if should_fail_commit_rename(&dst_path_str) {
drop(prepared_metadata_source);
Err(DiskError::other("test fail during metadata commit rename"))
} else {
os::rename_prepared_source_on_thread(
prepared_metadata_source,
&src_file_path_owned,
&dst_file_path_owned,
&skip_parent,
&publication_root_owned,
&guard,
)
};
if let Err(err) = rename_res {
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
}
if no_inline {
return Ok(FusedCommitAttempt::RestoreDataSource(guard, err));
if let Err(err) = os::rename_prepared_source_on_thread(
prepared_metadata_source,
&src_file_path_owned,
&dst_file_path_owned,
&skip_parent,
&publication_root_owned,
&guard,
) {
if no_inline && let Some((src_data_path, dst_data_path)) = has_data_dir_path_owned.as_ref() {
let _ = std::fs::rename(dst_data_path, src_data_path);
}
return Ok(FusedCommitAttempt::CommitErr(err));
}
if should_fail_after_metadata_commit(&dst_path_str) {
if no_inline {
rollback_inline_metadata_std(&dst_file_path_owned, None, staged_rollback_path.as_ref())?;
if let Some((_, new_data_path)) = has_data_dir_path_owned.as_ref() {
remove_dir_all_if_exists(new_data_path)?;
}
return Ok(FusedCommitAttempt::CommitErr(DiskError::Unexpected));
}
let err = rollback_inline_metadata_std(&dst_file_path_owned, None, staged_rollback_path.as_ref())
.err()
.unwrap_or_else(|| std::io::Error::other("test fail after metadata commit"));
return Ok(FusedCommitAttempt::CommitErr(DiskError::from(err)));
}
if crash_inject::should_crash_at(CrashPoint::RenameAfterMetaCommit, &dst_path_str) {
return Ok(FusedCommitAttempt::CommitErr(DiskError::Unexpected));
}
if sync_commit_metadata
&& let Err(err) = fsync_inline_commit_std(
&dst_file_path_owned,
&dst_volume_dir_owned,
destination_was_absent,
staged_rollback_path.as_ref(),
&guard,
)
&& let Err(err) =
fsync_inline_commit_std(&dst_file_path_owned, &dst_volume_dir_owned, destination_was_absent, None, &guard)
{
if no_inline {
if let Some((_, new_data_path)) = has_data_dir_path_owned.as_ref() {
remove_dir_all_if_exists(new_data_path)?;
}
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
if no_inline && let Some((src_data_path, dst_data_path)) = has_data_dir_path_owned.as_ref() {
let _ = std::fs::rename(dst_data_path, src_data_path);
}
return Ok(FusedCommitAttempt::CommitErr(DiskError::from(err)));
}
if let Some(backup_path) = staged_rollback_path.as_ref().map(StagedRollbackBackup::path) {
let _ = remove_file_if_exists(backup_path);
return Err(err);
}
if let Some(parent) = src_file_path_owned.parent() {
let _ = std::fs::remove_dir(parent);
}
Ok(FusedCommitAttempt::Committed(Box::new(RenameDataResp {
Ok(FusedCommitAttempt::Committed(RenameDataResp {
multipart_source: None,
old_data_dir: None,
rollback_data_dir: None,
@@ -1049,7 +909,7 @@ impl LocalDisk {
old_current_size,
old_current_source_checked,
old_current_source,
})))
}))
};
let attempt_res = if let Some(admission) = file_sync_admission.as_ref() {
@@ -1069,23 +929,7 @@ impl LocalDisk {
if let Some(receipt_mutation) = receipt_mutation.as_mut() {
receipt_mutation.complete();
}
return Ok(*resp);
}
Ok(FusedCommitAttempt::RestoreDataSource(directory_guard, err)) => {
let commit = RenameCommitContext {
directory_guard,
publication_root: &self.publication_root,
mutation_lease,
source_metadata: &src_file_path,
destination_metadata: &dst_file_path,
source_volume_dir: &src_volume_dir,
data_paths: has_data_dir_path.as_ref(),
};
commit.restore_published_data_source().await?;
for part_path in &invalidate_part_paths {
self.io_backend.invalidate_cached_fd(dst_volume, part_path).await;
}
return Err(err);
return Ok(resp);
}
Ok(FusedCommitAttempt::CommitErr(err)) | Err(err) => {
for part_path in &invalidate_part_paths {
@@ -1093,269 +937,11 @@ impl LocalDisk {
}
return Err(err);
}
#[cfg(test)]
Ok(FusedCommitAttempt::ResumePanic(panic_payload)) => {
std::panic::resume_unwind(panic_payload);
}
Ok(FusedCommitAttempt::Fallback(guard, dst_buf, restored_fi)) => {
fi = *restored_fi;
fi = restored_fi;
(guard, dst_buf)
}
}
} else if let Some(publication) = multipart_publication.as_ref()
&& no_inline
&& !preserve_upload_metadata
&& !publication.legacy
&& !fi_healing
&& !os::dst_dir_fsync_group_commit_enabled()
{
enum MultipartFusedCommitOutcome {
Committed(Box<RenameDataResp>),
AbortBeforeDataRename(DiskError),
RestoreDataSource(os::RenameCommitGuard, DiskError),
RestoreTargetAfterError(DiskError, bool),
}
let publication_owned = Arc::clone(publication);
let src_parent_owned = src_file_parent.to_path_buf();
let dst_parent_owned = dst_file_parent.to_path_buf();
let dst_file_path_owned = dst_file_path.clone();
let src_file_path_owned = src_file_path.clone();
let dst_volume_dir_owned = dst_volume_dir.clone();
let publication_root_owned = self.publication_root.clone();
let has_data_dir_path_owned = has_data_dir_path.clone();
let sync_data_shards = durability.syncs_data_shards();
let sync_commit_metadata = durability.syncs_commit_metadata();
let staging_cleanup = multipart_staging.as_ref().map(|s| s.cleanup_handle());
let dst_volume_str = dst_volume.to_string();
let dst_path_str = dst_path.to_string();
let fi_moved = fi;
#[cfg(test)]
let root_owned = self.root.clone();
let outcome = os::run_blocking_namespace_operation_with_sync(
mutation_lease.clone(),
true,
move || -> std::io::Result<MultipartFusedCommitOutcome> {
let (directory_guard_owned, has_dst_buf_owned) = match prepare_rename_commit_and_destination_metadata_std(
false,
&src_parent_owned,
&dst_parent_owned,
&dst_file_path_owned,
&dst_volume_dir_owned,
&publication_root_owned,
) {
Ok(res) => res,
Err(err) => return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(to_file_error(err).into())),
};
let mut xlmeta = FileMeta::new();
let mut dst_meta_unparsable = false;
if let Some(dst_buf) = has_dst_buf_owned.as_ref() {
if FileMeta::is_xl2_v1_format(dst_buf)
&& let Ok(nmeta) = FileMeta::load(dst_buf)
{
xlmeta = nmeta;
} else {
dst_meta_unparsable = true;
}
}
let skip_parent_owned = if has_dst_buf_owned.is_some() {
dst_parent_owned.clone()
} else {
dst_volume_dir_owned.clone()
};
let version_id = fi_moved.version_id.unwrap_or_default();
let (old_current_size, old_current_source_checked, old_current_source) = observe_old_current(
has_dst_buf_owned.is_some(),
dst_meta_unparsable,
&xlmeta,
&dst_volume_str,
&dst_path_str,
version_id,
);
let has_old_data_dir = xlmeta.find_unshared_data_dir_for_version(Some(version_id));
if let Some(old_data_dir) = has_old_data_dir.as_ref() {
let _ = xlmeta.data.remove_two(version_id, *old_data_dir);
}
if let Err(err) = xlmeta.add_version(fi_moved) {
return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(err.into()));
}
let version_signature = rename_data_versions_signature(&xlmeta);
let new_dst_buf_owned = match xlmeta.marshal_msg() {
Ok(buf) => buf,
Err(err) => return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(err.into())),
};
let prepared_metadata_source = match publication_owned.prepare_and_install_head_on_thread(
&new_dst_buf_owned,
has_dst_buf_owned.as_deref(),
&dst_file_path_owned,
&directory_guard_owned,
) {
Ok(source) => source,
Err(err) => return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(err)),
};
#[cfg(test)]
let probe_active = os::file_sync_probe::is_active();
#[cfg(not(test))]
let probe_active = false;
// When presynced, shard payloads were synced during UploadPart and
// prepare_and_install_head_on_thread just fsynced src_data_path after
// writing RETIREMENT_ANCHOR.
if sync_data_shards
&& let Some((src_data_path, _)) = has_data_dir_path_owned.as_ref()
&& (probe_active || !super::take_presynced_tmp_data_dir(src_data_path))
&& let Err(err) = os::sync_dir_files_std(src_data_path)
&& err.kind() != ErrorKind::NotFound
{
return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(to_file_error(err).into()));
}
let directory_guard_owned = match super::remove_dst_base_before_commit_std(
&dst_path_str,
directory_guard_owned,
&src_parent_owned,
&dst_parent_owned,
&dst_volume_dir_owned,
&publication_root_owned,
) {
Ok(guard) => guard,
Err(err) => return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(err)),
};
if should_remove_staged_meta_before_commit(&dst_path_str) {
drop(prepared_metadata_source);
if let Err(err) = std::fs::remove_file(&src_file_path_owned) {
return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(to_file_error(err).into()));
}
return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(DiskError::FileNotFound));
}
if let Some((src_data_path, dst_data_path)) = has_data_dir_path_owned.as_ref() {
let Some(src_parent) = src_data_path.parent() else {
return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(DiskError::InvalidPath));
};
let payload_guard = match os::multipart_payload_commit_guard_std(&directory_guard_owned, src_parent) {
Ok(guard) => guard,
Err(err) => return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(to_file_error(err).into())),
};
if let Err(err) = os::rename_all_with_commit_guard_on_thread(src_data_path, dst_data_path, &payload_guard)
{
return Ok(MultipartFusedCommitOutcome::RestoreDataSource(directory_guard_owned, err));
}
#[cfg(test)]
run_rename_data_after_first_publication(&root_owned, &dst_volume_str, &dst_path_str);
if crash_inject::should_crash_at(CrashPoint::RenameAfterDataRename, &dst_path_str) {
return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(DiskError::Unexpected));
}
if should_fail_before_old_metadata_backup(&dst_path_str) {
return Ok(MultipartFusedCommitOutcome::RestoreDataSource(
directory_guard_owned,
DiskError::Unexpected,
));
}
if crash_inject::should_crash_at(CrashPoint::RenameAfterBackupBeforeMetaCommit, &dst_path_str) {
return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(DiskError::Unexpected));
}
}
let rename_res = if should_fail_commit_rename(&dst_path_str) {
drop(prepared_metadata_source);
Err(DiskError::other("test fail during metadata commit rename"))
} else {
os::rename_prepared_source_on_thread(
prepared_metadata_source,
&src_file_path_owned,
&dst_file_path_owned,
&skip_parent_owned,
&publication_root_owned,
&directory_guard_owned,
)
};
if let Err(err) = rename_res {
return Ok(MultipartFusedCommitOutcome::RestoreDataSource(directory_guard_owned, err));
}
if should_fail_after_metadata_commit(&dst_path_str) {
return Ok(MultipartFusedCommitOutcome::RestoreTargetAfterError(DiskError::Unexpected, false));
}
if crash_inject::should_crash_at(CrashPoint::RenameAfterMetaCommit, &dst_path_str) {
return Ok(MultipartFusedCommitOutcome::AbortBeforeDataRename(DiskError::Unexpected));
}
if sync_commit_metadata {
if let Some(parent) = dst_file_path_owned.parent()
&& let Err(err) = os::fsync_commit_directory_limited_std(parent, &directory_guard_owned)
{
return Ok(MultipartFusedCommitOutcome::RestoreTargetAfterError(to_file_error(err).into(), true));
}
if has_dst_buf_owned.is_none() {
let mut ancestor = dst_file_path_owned.parent().and_then(|object_dir| object_dir.parent());
while let Some(dir) = ancestor {
if !dir.starts_with(&dst_volume_dir_owned) {
break;
}
if let Err(err) = os::fsync_dir_limited_std(dir) {
return Ok(MultipartFusedCommitOutcome::RestoreTargetAfterError(
to_file_error(err).into(),
true,
));
}
if dir == dst_volume_dir_owned.as_path() {
break;
}
ancestor = dir.parent();
}
}
}
match publication_owned.finish_on_thread(sync_commit_metadata, staging_cleanup) {
Ok(result) => Ok(MultipartFusedCommitOutcome::Committed(Box::new(RenameDataResp {
multipart_source: Some(Box::new(result)),
old_data_dir: has_old_data_dir,
rollback_data_dir: None,
cleanup_data_dir: has_old_data_dir,
sign: version_signature,
old_current_size,
old_current_source_checked,
old_current_source,
}))),
Err(err) => Ok(MultipartFusedCommitOutcome::RestoreTargetAfterError(err, false)),
}
},
)
.await
.map_err(to_file_error)
.map_err(DiskError::from)?;
match outcome {
MultipartFusedCommitOutcome::Committed(resp) => {
for part_path in &invalidate_part_paths {
self.io_backend.invalidate_cached_fd(dst_volume, part_path).await;
}
return Ok(*resp);
}
MultipartFusedCommitOutcome::AbortBeforeDataRename(err) => return Err(err),
MultipartFusedCommitOutcome::RestoreDataSource(directory_guard, err) => {
let commit = RenameCommitContext {
directory_guard,
publication_root: &self.publication_root,
mutation_lease,
source_metadata: &src_file_path,
destination_metadata: &dst_file_path,
source_volume_dir: &src_volume_dir,
data_paths: has_data_dir_path.as_ref(),
};
commit.restore_published_data_source().await?;
return Err(err);
}
MultipartFusedCommitOutcome::RestoreTargetAfterError(err, invalidate_before_return) => {
if !invalidate_before_return {
for part_path in &invalidate_part_paths {
self.io_backend.invalidate_cached_fd(dst_volume, part_path).await;
}
}
pin_multipart_future(|| publication.restore_target_after_error(mutation_lease.clone())).await?;
if invalidate_before_return {
for part_path in &invalidate_part_paths {
self.io_backend.invalidate_cached_fd(dst_volume, part_path).await;
}
}
return Err(err);
}
}
} else {
prepare_rename_commit_and_destination_metadata(
!no_inline && !preserve_upload_metadata,
@@ -1479,81 +1065,71 @@ impl LocalDisk {
// after the commit rename, and a crash before the rename means the
// PUT was never acknowledged. A metadata commit: relaxed tiers
// leave it to the page cache.
#[cfg(all(test, target_os = "linux"))]
let probe_active = os::file_sync_probe::is_active();
#[cfg(all(not(test), target_os = "linux"))]
let probe_active = false;
if let Some(publication) = multipart_publication.as_ref() {
pin_multipart_future(|| publication.prepare_metadata(&new_dst_buf, has_dst_buf.as_deref())).await?;
}
let tmp_meta_sync = if durability.syncs_commit_metadata() || multipart_publication.is_some() {
SyncMode::FileOnly
} else {
SyncMode::None
};
// The tmp xl.meta write and the shard-file fdatasync are independent
// (disjoint paths) and both only need to be durable before the commit
// renames below, so run them concurrently to drop a blocking
// round-trip from the PUT commit critical path (rustfs/backlog#922
// step 2). The "contents durable -> rename -> dst dir fsync" ordering
// is unchanged — both futures complete before any rename — which the
// rename_data crash-consistency harness (backlog#935) exercises.
//
// Shard durability: once rename_data succeeds the write is
// acknowledged, so data must not live only in the page cache.
// Multipart parts were already synced during rename_part, so their
// fdatasync here is a cheap no-op. A missing source dir is left for the
// rename below to report through the existing rollback path. Payload
// durability is kept by both strict and relaxed.
let tmp_meta_write = {
let src_file_path = multipart_publication
.as_ref()
.map_or_else(|| commit.source_metadata.to_path_buf(), |publication| publication.prepared_path.clone());
let dst_file_path = dst_file_path.clone();
let owned_directory_guard = commit.directory_guard.clone();
let owned_mutation_lease = commit.mutation_lease.clone();
async move {
os::run_blocking_namespace_operation(owned_mutation_lease, move || {
#[cfg(test)]
run_owned_file_before_open(&src_file_path);
let mut prepared_metadata_source = os::create_prepared_rename_source_with_commit_guard(
&src_file_path,
&dst_file_path,
&owned_directory_guard,
)?;
prepared_metadata_source.write_all(&new_dst_buf, tmp_meta_sync != SyncMode::None)?;
Ok(prepared_metadata_source)
})
.await
.map_err(to_file_error)
.map_err(DiskError::from)
}
};
let shard_sync = async {
if durability.syncs_data_shards()
&& let Some((src_data_path, _)) = has_data_dir_path.as_ref()
&& let Err(err) = os::sync_dir_files_with_limiter(src_data_path, self.file_sync_permits.clone()).await
&& err.kind() != ErrorKind::NotFound
{
#[cfg(target_os = "linux")]
if !probe_active && super::take_presynced_tmp_data_dir(src_data_path) {
if multipart_publication.is_none()
&& let Err(err) = os::fsync_dir_with_owner(src_data_path, Some(commit.mutation_lease.clone())).await
&& err.kind() != ErrorKind::NotFound
{
return Err::<(), DiskError>(to_file_error(err).into());
}
return Ok(());
}
if let Err(err) = os::sync_dir_files_with_limiter(src_data_path, self.file_sync_permits.clone()).await
&& err.kind() != ErrorKind::NotFound
{
return Err::<(), DiskError>(to_file_error(err).into());
}
return Err::<(), DiskError>(to_file_error(err).into());
}
Ok(())
};
let prepared_metadata_source = if let Some(publication) = multipart_publication.as_ref() {
let dst_file_path_ref = &dst_file_path;
let dir_guard_ref = &commit.directory_guard;
let lease_clone = commit.mutation_lease.clone();
let prepare_fut = pin_multipart_future(|| {
publication.prepare_and_install_head(
&new_dst_buf,
has_dst_buf.as_deref(),
dst_file_path_ref,
dir_guard_ref,
lease_clone,
)
});
let (tmp_meta_res, shard_sync_res) = tokio::join!(prepare_fut, shard_sync);
let prepared_metadata_source = tmp_meta_res?;
shard_sync_res?;
prepared_metadata_source
} else {
let tmp_meta_sync = if durability.syncs_commit_metadata() {
SyncMode::FileOnly
} else {
SyncMode::None
};
let tmp_meta_write = {
let src_file_path = commit.source_metadata.to_path_buf();
let dst_file_path = dst_file_path.clone();
let owned_directory_guard = commit.directory_guard.clone();
let owned_mutation_lease = commit.mutation_lease.clone();
async move {
os::run_blocking_namespace_operation(owned_mutation_lease, move || {
#[cfg(test)]
run_owned_file_before_open(&src_file_path);
let mut prepared_metadata_source = os::create_prepared_rename_source_with_commit_guard(
&src_file_path,
&dst_file_path,
&owned_directory_guard,
)?;
prepared_metadata_source.write_all(&new_dst_buf, tmp_meta_sync != SyncMode::None)?;
Ok(prepared_metadata_source)
})
.await
.map_err(to_file_error)
.map_err(DiskError::from)
}
};
let (tmp_meta_res, shard_sync_res) = tokio::join!(tmp_meta_write, shard_sync);
let prepared_metadata_source = tmp_meta_res?;
shard_sync_res?;
prepared_metadata_source
let (tmp_meta_res, shard_sync_res) = tokio::join!(tmp_meta_write, shard_sync);
// Surface a tmp-meta failure first (its prior serial position), then a
// shard-sync failure; either aborts before any rename, exactly as the
// sequential version did.
let prepared_metadata_source = tmp_meta_res?;
shard_sync_res?;
let prepared_metadata_source = match multipart_publication.as_ref() {
Some(publication) => pin_multipart_future(|| publication.install_head(prepared_metadata_source)).await?,
None => prepared_metadata_source,
};
commit.directory_guard = remove_dst_base_before_commit(
dst_path,
@@ -1893,22 +1469,13 @@ impl LocalDisk {
}
let multipart_source = match multipart_publication.as_ref() {
Some(publication) => {
let target_already_synced = durability.syncs_commit_metadata();
let staging_cleanup = multipart_staging.as_ref().map(|s| s.cleanup_handle());
match pin_multipart_future(|| {
publication.finish(commit.mutation_lease.clone(), target_already_synced, staging_cleanup)
})
.await
{
Ok(result) => Some(Box::new(result)),
Err(error) => {
pin_multipart_future(|| publication.restore_target_after_error(commit.mutation_lease.clone()))
.await?;
return Err(error);
}
Some(publication) => match pin_multipart_future(|| publication.finish(commit.mutation_lease.clone())).await {
Ok(result) => Some(Box::new(result)),
Err(error) => {
pin_multipart_future(|| publication.restore_target_after_error(commit.mutation_lease.clone())).await?;
return Err(error);
}
}
},
None => None,
};
Ok(RenameDataResp {
@@ -2108,8 +1675,13 @@ impl LocalDisk {
#[cfg(not(windows))]
let backup_path = {
let _backup = os::StageTimer::new(native_timing, "pool_meta_native_backup_create");
create_local_inline_rollback_backup(&dst, &src, old_metadata, sync)?
create_local_inline_rollback_backup(&dst, &src, old_metadata)?
};
#[cfg(not(windows))]
if sync {
let _backup_sync = os::StageTimer::new(native_timing, "pool_meta_native_backup_file_sync");
std::fs::File::open(&backup_path)?.sync_data()?;
}
staged_rollback_path = Some(StagedRollbackBackup(backup_path));
}
+136 -498
View File
@@ -1000,25 +1000,6 @@ pub(super) fn inspection_with_markers(
Ok((inspection, has_physical_data_dirs_or_markers))
}
type PreparedDeleteReceiptEntry = (DeleteRecoveryJournal, bool);
static PREPARED_DELETE_RECEIPTS: std::sync::LazyLock<parking_lot::Mutex<HashMap<PathBuf, PreparedDeleteReceiptEntry>>> =
std::sync::LazyLock::new(|| parking_lot::Mutex::new(HashMap::new()));
fn put_prepared_delete_receipt(rollback: PathBuf, entry: PreparedDeleteReceiptEntry) {
let mut map = PREPARED_DELETE_RECEIPTS.lock();
if map.len() >= 4096
&& let Some(evict_key) = map.keys().next().cloned()
{
map.remove(&evict_key);
}
map.insert(rollback, entry);
}
fn take_prepared_delete_receipt(rollback: &Path) -> Option<PreparedDeleteReceiptEntry> {
PREPARED_DELETE_RECEIPTS.lock().remove(rollback)
}
impl LocalDisk {
// Native MPU source closure has its own conditional protocol. Generic
// object recovery must never restore its head or acquire SourceRW under
@@ -1542,9 +1523,6 @@ impl LocalDisk {
owner: Option<Arc<dyn Send + Sync>>,
) -> Result<DeleteRecoveryInspection> {
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if let Ok(root) = self.io_get_object_path(volume, object) {
let _ = take_prepared_delete_receipt(&root.join(transaction.to_string()));
}
let (mut observed, mut has_data_dir_work) = self
.delete_recovery_inspect_owned_with_markers(volume, object, transaction, owner.clone())
@@ -1933,14 +1911,8 @@ impl LocalDisk {
return Err(DiskError::DiskOngoingReq);
}
let bucket_path = self.io_get_bucket_path(volume)?;
let mut file_leases = Vec::with_capacity(5);
for name in [
STORAGE_FORMAT_FILE_BACKUP,
DELETE_MARKER_ROLLBACK_FILE,
TARGET,
PREPARED,
FINISHED,
] {
let mut file_leases = Vec::with_capacity(3);
for name in [TARGET, PREPARED, FINISHED] {
let path = rollback.join(name);
let lease = os::acquire_namespace_mutation_lease_with_owner(&path, owner.clone()).await;
file_leases.push((path, lease));
@@ -2160,151 +2132,119 @@ impl LocalDisk {
Err(err) if err.kind() == ErrorKind::NotFound => return Ok(Ok(None)),
Err(err) => return Err(err),
};
type PreparedDeleteTuple = (Option<Uuid>, Option<Vec<u8>>, Vec<u8>, DeleteRecoveryJournal, bool);
let (old_dir, target, journal_raw, journal, sole_transaction) =
match (|| -> Result<Option<PreparedDeleteTuple>> {
let mut meta = FileMeta::load(&buf)?;
let mut deleted_headers: Vec<_> = meta
.versions
.iter()
.map(|v| (v.header.version_id, std::mem::discriminant(&v.header.version_type)))
.collect();
let original_versions: Vec<_> = meta
.versions
.iter()
.map(|v| generation(v.into_fileinfo(&volume_str, &object_str, true)?))
.collect::<Result<_>>()?;
let old_dir = meta.delete_version(&fi_clone)?;
if old_dir.is_some_and(|dir| dir == transaction || dir.is_nil()) {
return Err(DiskError::FileCorrupt);
let (old_dir, journal_raw) = match (|| -> Result<Option<(Option<Uuid>, Vec<u8>)>> {
let mut meta = FileMeta::load(&buf)?;
let deleted_versions: Vec<_> = meta
.versions
.iter()
.map(|v| v.header.version_id.unwrap_or_default())
.collect();
let original_versions: Vec<_> = meta
.versions
.iter()
.map(|v| generation(v.into_fileinfo(&volume_str, &object_str, true)?))
.collect::<Result<_>>()?;
let old_dir = meta.delete_version(&fi_clone)?;
if old_dir.is_some_and(|dir| dir == transaction || dir.is_nil()) {
return Err(DiskError::FileCorrupt);
}
if let Some(uuid) = old_dir {
meta.data.remove(vec![fi_clone.version_id.unwrap_or_default(), uuid])?;
let old_path = path_join(&[file_path.as_path(), Path::new(uuid.to_string().as_str())]);
check_path_length(old_path.to_string_lossy().as_ref())?;
if old_path.join(STORAGE_FORMAT_FILE).exists() {
return Ok(None);
}
if let Some(uuid) = old_dir {
meta.data.remove(vec![fi_clone.version_id.unwrap_or_default(), uuid])?;
let old_path = path_join(&[file_path.as_path(), Path::new(uuid.to_string().as_str())]);
check_path_length(old_path.to_string_lossy().as_ref())?;
if old_path.join(STORAGE_FORMAT_FILE).exists() {
return Ok(None);
}
if !meta.versions.is_empty() {
return Ok(None);
}
let mut predecessors = BTreeSet::new();
if let Ok(entries) = std::fs::read_dir(&file_path) {
for (visited, entry) in entries.enumerate() {
if visited >= 65536 {
return Err(DiskError::DiskOngoingReq);
}
let entry = entry.map_err(to_file_error)?;
let Some(id) = entry
.file_name()
.to_str()
.and_then(|name| Uuid::parse_str(name).ok())
.filter(|id| !id.is_nil() && *id != transaction)
else {
continue;
};
if !entry.file_type().map_err(to_file_error)?.is_dir() {
return Err(DiskError::InvalidPath);
}
let prepared = regular_bytes(&entry.path().join(PREPARED))?;
let receipt = regular_bytes(&entry.path().join(DELETE_QUORUM_COMMITTED_FILE))?;
let mut records = Vec::new();
if let Some(bytes) = prepared {
records.push(
serde_json::from_slice::<DeleteRecoveryJournal>(&bytes)
.map_err(|_| DiskError::FileCorrupt)?,
);
}
if let Some(bytes) = receipt.filter(|bytes| !bytes.is_empty()) {
records.push(
serde_json::from_slice::<CommitReceipt>(&bytes)
.map_err(|_| DiskError::FileCorrupt)?
.journal,
);
}
for record in records {
if record.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
|| record.volume != volume_str
|| record.object != object_str
|| record.transaction != id
|| incarnation.zip(record.bucket_incarnation).is_some_and(|(a, b)| a != b)
|| record.predecessors.iter().any(|before| before.is_nil() || *before == id)
{
return Err(DiskError::FileCorrupt);
}
predecessors.insert(id);
}
}
let surviving: HashSet<_> = meta
.versions
.iter()
.map(|v| (v.header.version_id, std::mem::discriminant(&v.header.version_type)))
.collect();
deleted_headers.retain(|header| !surviving.contains(header));
let deleted_versions = deleted_headers.into_iter().map(|(id, _)| id.unwrap_or_default()).collect();
let target = (!meta.versions.is_empty()).then(|| meta.marshal_msg()).transpose()?;
let mut predecessors = BTreeSet::new();
let mut sole_transaction = true;
if let Ok(entries) = std::fs::read_dir(&file_path) {
for (visited, entry) in entries.enumerate() {
if visited >= 65536 {
return Err(DiskError::DiskOngoingReq);
}
let entry = entry.map_err(to_file_error)?;
let file_name = entry.file_name();
let Some(name_str) = file_name.to_str() else {
sole_transaction = false;
continue;
};
let Some(id) = Uuid::parse_str(name_str).ok().filter(|id| !id.is_nil() && *id != transaction)
else {
continue;
};
if !entry.file_type().map_err(to_file_error)?.is_dir() {
return Err(DiskError::InvalidPath);
}
let dir_path = entry.path();
let prepared = regular_bytes(&dir_path.join(PREPARED))?;
let receipt = regular_bytes(&dir_path.join(DELETE_QUORUM_COMMITTED_FILE))?;
if sole_transaction && name_str == id.hyphenated().to_string() && old_dir != Some(id) {
let is_surviving_data_dir = original_versions.iter().any(|v| v.data_dir == Some(id))
&& prepared.is_none()
&& receipt.is_none()
&& !dir_path.join(STORAGE_FORMAT_FILE_BACKUP).exists();
if !is_surviving_data_dir {
sole_transaction = false;
}
}
let mut records = Vec::new();
if let Some(bytes) = prepared {
records.push(
serde_json::from_slice::<DeleteRecoveryJournal>(&bytes)
.map_err(|_| DiskError::FileCorrupt)?,
);
}
if let Some(bytes) = receipt.filter(|bytes| !bytes.is_empty()) {
records.push(
serde_json::from_slice::<CommitReceipt>(&bytes)
.map_err(|_| DiskError::FileCorrupt)?
.journal,
);
}
for record in records {
if record.protocol_version != DELETE_RECOVERY_PROTOCOL_VERSION
|| record.volume != volume_str
|| record.object != object_str
|| record.transaction != id
|| incarnation.zip(record.bucket_incarnation).is_some_and(|(a, b)| a != b)
|| record.predecessors.iter().any(|before| before.is_nil() || *before == id)
{
return Err(DiskError::FileCorrupt);
}
predecessors.insert(id);
}
}
}
sole_transaction &= predecessors.is_empty();
let mut sparse = fi_clone;
sparse.data = None;
let journal = DeleteRecoveryJournal {
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
volume: volume_str,
object: object_str,
transaction,
bucket_incarnation: incarnation,
original_digest: Some(digest(&buf)),
target_digest: target.as_deref().map(digest),
data_dirs: old_dir.into_iter().collect(),
deleted_versions,
requests: vec![sparse],
original_versions,
predecessors: predecessors.into_iter().collect(),
};
let raw = journal_bytes(&journal)?;
Ok(Some((old_dir, target, raw, journal, sole_transaction)))
})() {
Ok(Some(tuple)) => tuple,
Ok(None) => return Ok(Ok(None)),
Err(err) => return Ok(Err(err)),
}
let mut sparse = fi_clone;
sparse.data = None;
let journal = DeleteRecoveryJournal {
protocol_version: DELETE_RECOVERY_PROTOCOL_VERSION,
volume: volume_str,
object: object_str,
transaction,
bucket_incarnation: incarnation,
original_digest: Some(digest(&buf)),
target_digest: None,
data_dirs: old_dir.into_iter().collect(),
deleted_versions,
requests: vec![sparse],
original_versions,
predecessors: predecessors.into_iter().collect(),
};
Ok(Some((old_dir, journal_bytes(&journal)?)))
})() {
Ok(Some(pair)) => pair,
Ok(None) => return Ok(Ok(None)),
Err(err) => return Ok(Err(err)),
};
std::fs::create_dir_all(&staging_root)?;
let staged = tempfile::Builder::new().prefix("delete-prep-").tempdir_in(&staging_root)?;
let staged_dir = staged.path().to_path_buf();
let backup_path = staged_dir.join(STORAGE_FORMAT_FILE_BACKUP);
if std::fs::hard_link(&xl_path, &backup_path).is_err() {
let mut file = std::fs::File::create(&backup_path)?;
std::io::Write::write_all(&mut file, &buf)?;
if sync && !buf.is_empty() {
let staged = tempfile::Builder::new()
.prefix("delete-preparation-")
.tempdir_in(&staging_root)?;
for (name, bytes) in [
(STORAGE_FORMAT_FILE_BACKUP, buf.as_slice()),
(PREPARED, journal_raw.as_slice()),
] {
let mut file = std::fs::File::create(staged.path().join(name))?;
std::io::Write::write_all(&mut file, bytes)?;
if sync && !bytes.is_empty() {
file.sync_data()?;
}
}
let mut written_files = Vec::with_capacity(2);
for (name, bytes) in [(TARGET, target.as_deref()), (PREPARED, Some(journal_raw.as_slice()))] {
let Some(bytes) = bytes else {
continue;
};
let mut file = std::fs::File::create(staged_dir.join(name))?;
std::io::Write::write_all(&mut file, bytes)?;
if sync && !bytes.is_empty() {
written_files.push(file);
}
}
for file in written_files {
file.sync_data()?;
}
if sync {
os::fsync_dir_std(&staged_dir)?;
os::fsync_dir_std(staged.path())?;
}
match std::fs::symlink_metadata(&destination) {
Ok(metadata) if metadata.is_dir() => match std::fs::remove_dir(&destination) {
@@ -2317,62 +2257,28 @@ impl LocalDisk {
Err(err) if err.kind() == ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
let staged_dir = staged.keep();
if let Err(err) = std::fs::rename(&staged_dir, &destination) {
let _ = std::fs::remove_dir_all(&staged_dir);
return Err(err);
if let Some(parent) = destination.parent() {
std::fs::create_dir_all(parent)?;
}
let old_data_path = old_dir.map(|uuid| file_path.join(uuid.to_string())).filter(|p| p.is_dir());
if sync && let Err(err) = os::fsync_dir_std(&file_path) {
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
os::record_native_publication(&destination);
std::fs::rename(staged.path(), &destination)?;
if sync {
os::fsync_dir_std(&file_path)?;
}
let mut reserved_data_path = None;
if let Some(old_path) = old_data_path {
let reserved_marker = old_path.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
let reserve_res = (|| -> std::io::Result<()> {
if let Some(uuid) = old_dir {
let old_path = file_path.join(uuid.to_string());
if old_path.is_dir() {
let reserved_marker = old_path.join(format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"));
let marker = std::fs::File::create(&reserved_marker)?;
if sync {
marker.sync_all()?;
os::fsync_dir_std(&old_path)?;
}
Ok(())
})();
if let Err(err) = reserve_res {
let _ = std::fs::remove_file(&reserved_marker);
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
reserved_data_path = Some((old_path, reserved_marker));
}
reserved_data_path = Some((old_path, reserved_marker));
}
if let Some(ref target_bytes) = target {
let target_in_rollback = destination.join(TARGET);
let linked_xl = staging_root.join(format!("delete-xl-{transaction}"));
let _ = std::fs::remove_file(&linked_xl);
let write_res = if std::fs::hard_link(&target_in_rollback, &linked_xl).is_ok() {
let rename_res = std::fs::rename(&linked_xl, &xl_path);
if rename_res.is_err() {
let _ = std::fs::remove_file(&linked_xl);
}
rename_res
} else {
(|| -> std::io::Result<()> {
let mut staged_xl = tempfile::NamedTempFile::new_in(&staging_root)?;
std::io::Write::write_all(staged_xl.as_file_mut(), target_bytes)?;
if sync && !target_bytes.is_empty() {
staged_xl.as_file().sync_data()?;
}
std::fs::rename(staged_xl.path(), &xl_path)
})()
};
if let Err(err) = write_res {
if let Some((_, reserved_marker)) = &reserved_data_path {
let _ = std::fs::remove_file(reserved_marker);
}
let _ = std::fs::remove_dir_all(&destination);
return Err(err);
}
} else if let Err(err) = std::fs::remove_file(&xl_path)
if let Err(err) = std::fs::remove_file(&xl_path)
&& err.kind() != ErrorKind::NotFound
{
if let Some((_, reserved_marker)) = &reserved_data_path {
@@ -2393,7 +2299,6 @@ impl LocalDisk {
Err(err) => return Err(err),
}
}
put_prepared_delete_receipt(destination, (journal, sole_transaction));
Ok(Ok(Some(old_dir)))
},
)
@@ -2439,26 +2344,6 @@ impl LocalDisk {
lease,
sync,
move || -> std::io::Result<Result<Option<(DeleteRecoveryJournal, bool, bool)>>> {
if let Some((journal, sole_transaction)) = take_prepared_delete_receipt(&rollback)
&& rollback.join(PREPARED).is_file()
{
let receipt = CommitReceipt {
journal: journal.clone(),
finished: false,
};
let receipt_bytes = serde_json::to_vec(&receipt).map_err(std::io::Error::other)?;
std::fs::create_dir_all(&staging_root)?;
let mut staged = tempfile::NamedTempFile::new_in(&staging_root)?;
std::io::Write::write_all(staged.as_file_mut(), &receipt_bytes)?;
if sync && !receipt_bytes.is_empty() {
staged.as_file().sync_data()?;
}
std::fs::rename(staged.path(), &receipt_path)?;
if sync {
os::fsync_dir_std(&rollback)?;
}
return Ok(Ok(Some((journal, sole_transaction, true))));
}
let prepared = match regular_bytes(&rollback.join(PREPARED)) {
Ok(Some(bytes)) => bytes,
Ok(None) => return Ok(Ok(None)),
@@ -2476,18 +2361,12 @@ impl LocalDisk {
{
return Ok(Err(DiskError::FileCorrupt));
}
let target_satisfied = match journal.target_digest {
None => match regular_file_exists(&root.join(STORAGE_FORMAT_FILE)) {
Ok(exists) => !exists,
Err(e) => return Ok(Err(e)),
},
Some(expected_target) => match regular_bytes(&root.join(STORAGE_FORMAT_FILE)) {
Ok(Some(bytes)) => digest(&bytes) == expected_target,
Ok(None) => false,
Err(e) => return Ok(Err(e)),
},
let current_exists = match regular_file_exists(&root.join(STORAGE_FORMAT_FILE)) {
Ok(exists) => exists,
Err(e) => return Ok(Err(e)),
};
if !target_satisfied {
let unversioned_target_absent = journal.target_digest.is_none() && !current_exists;
if !unversioned_target_absent {
return Ok(Ok(None));
}
let mut sole_transaction = journal.predecessors.is_empty();
@@ -2504,15 +2383,8 @@ impl LocalDisk {
&& id != transaction
&& !journal.data_dirs.contains(&id)
{
let dir_path = entry.path();
let is_surviving_data_dir = journal.original_versions.iter().any(|v| v.data_dir == Some(id))
&& !dir_path.join(PREPARED).exists()
&& !dir_path.join(DELETE_QUORUM_COMMITTED_FILE).exists()
&& !dir_path.join(STORAGE_FORMAT_FILE_BACKUP).exists();
if !is_surviving_data_dir {
sole_transaction = false;
break;
}
sole_transaction = false;
break;
}
}
}
@@ -2531,11 +2403,12 @@ impl LocalDisk {
if sync && !receipt_bytes.is_empty() {
staged.as_file().sync_data()?;
}
os::record_native_publication(&receipt_path);
std::fs::rename(staged.path(), &receipt_path)?;
if sync {
os::fsync_dir_std(&rollback)?;
}
Ok(Ok(Some((journal, sole_transaction, target_satisfied))))
Ok(Ok(Some((journal, sole_transaction, unversioned_target_absent))))
},
)
.await
@@ -2587,15 +2460,12 @@ impl LocalDisk {
}
let bucket_path = self.io_get_bucket_path(volume)?;
let staging_root = self.io_get_bucket_path(RUSTFS_META_TMP_BUCKET)?;
let root = self.io_get_object_path(volume, object)?;
let rollback = root.join(transaction.to_string());
let receipt_path = rollback.join(DELETE_QUORUM_COMMITTED_FILE);
let sync = effective_durability(volume).syncs_commit_metadata();
let data_dirs = journal.data_dirs.clone();
let has_target = journal.target_digest.is_some();
let journal_for_receipt = journal.clone();
let lease = os::acquire_namespace_mutation_lease_with_owner(&receipt_path, owner).await;
let data_dirs = journal.data_dirs.clone();
let prune_paths: Vec<_> = rollback
.ancestors()
.take_while(|dir| *dir != bucket_path.as_path() && dir.starts_with(&bucket_path))
@@ -2606,13 +2476,11 @@ impl LocalDisk {
let committed = format!("{DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let recovery_committed = format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let reserved = format!("{RESERVED_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}");
let mut had_physical_data_dirs = false;
for data_dir in &data_dirs {
for base in [&root, &rollback] {
let dir_path = base.join(data_dir.to_string());
match std::fs::read_dir(&dir_path) {
Ok(entries) => {
had_physical_data_dirs = true;
let mut removable = Vec::new();
let mut preserve_dir = false;
for entry in entries {
@@ -2653,45 +2521,23 @@ impl LocalDisk {
}
}
}
if had_physical_data_dirs && sync && root.is_dir() {
if !data_dirs.is_empty() && sync && root.is_dir() {
os::fsync_dir_std(&root)?;
}
for name in [STORAGE_FORMAT_FILE_BACKUP, DELETE_MARKER_ROLLBACK_FILE] {
for name in [
STORAGE_FORMAT_FILE_BACKUP,
DELETE_MARKER_ROLLBACK_FILE,
TARGET,
PREPARED,
FINISHED,
] {
ignore_not_found(std::fs::remove_file(rollback.join(name)))?;
}
if has_target || had_physical_data_dirs {
let finished_receipt = CommitReceipt {
journal: journal_for_receipt,
finished: true,
};
let receipt_bytes = serde_json::to_vec(&finished_receipt).map_err(std::io::Error::other)?;
std::fs::create_dir_all(&staging_root)?;
let mut staged_receipt = tempfile::NamedTempFile::new_in(&staging_root)?;
std::io::Write::write_all(staged_receipt.as_file_mut(), &receipt_bytes)?;
if sync && !receipt_bytes.is_empty() {
staged_receipt.as_file().sync_data()?;
}
std::fs::rename(staged_receipt.path(), &receipt_path)?;
let _ = std::fs::File::create(rollback.join(FINISHED))?;
if sync {
os::fsync_dir_std(&rollback)?;
}
}
for name in [TARGET, PREPARED, FINISHED] {
let path = rollback.join(name);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &path);
ignore_not_found(std::fs::remove_file(path))?;
}
if sync {
os::fsync_dir_std(&rollback)?;
}
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &receipt_path);
ignore_not_found(std::fs::remove_file(&receipt_path))?;
for dir_path in &prune_paths {
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, dir_path);
if let Err(err) = std::fs::remove_dir(dir_path) {
if is_dir_not_empty_error(&err) {
break;
@@ -6438,214 +6284,6 @@ mod tests {
b"outside bytes"
);
}
#[tokio::test]
async fn delete_recovery_fused_versioned_delete_marker_and_publish_fast_path() {
let (_temp, disk, info, _) = fixture().await;
let root = disk.io_get_object_path("recovery", "object").expect("object root");
let transaction = Uuid::new_v4();
let mut marker = FileInfo {
name: "object".to_owned(),
version_id: Some(Uuid::new_v4()),
deleted: true,
mark_deleted: true,
mod_time: info.mod_time.map(|t| t + time::Duration::seconds(1)),
..Default::default()
};
let opts = DeleteOptions {
old_data_dir: Some(transaction),
recoverable_delete: true,
..Default::default()
};
let fused = disk
.try_fused_recoverable_delete_version("recovery", "object", &marker, &opts, transaction, None)
.await
.expect("fused delete marker preparation must succeed");
assert_eq!(fused, Some(()), "versioned delete marker must use fused path");
assert!(root.join(transaction.to_string()).join(TARGET).is_file());
assert!(root.join(STORAGE_FORMAT_FILE).is_file());
let published = disk
.try_publish_prepared_delete_local("recovery", "object", transaction, None)
.await
.expect("local publish must succeed")
.expect("versioned target must satisfy local publish");
assert!(published.1, "sole transaction must be true");
assert!(published.2, "target_satisfied must be true");
let finalized = disk
.try_finish_and_finalize_prepared_delete_local("recovery", "object", transaction, &published.0, None)
.await
.expect("local finalize must succeed");
assert!(finalized);
assert!(!root.join(transaction.to_string()).exists());
let final_meta = FileMeta::load(&fs::read(root.join(STORAGE_FORMAT_FILE)).await.expect("read final xl.meta"))
.expect("decode final xl.meta");
assert_eq!(final_meta.versions.len(), 2);
assert_eq!(final_meta.versions[0].header.version_id, marker.version_id);
marker.deleted = false;
}
#[tokio::test]
async fn delete_recovery_fused_versioned_delete_with_surviving_version_and_interrupted_target_cleanup() {
let (_temp, disk, mut v1, _) = fixture().await;
let root = disk.io_get_object_path("recovery", "object").expect("object path");
// Configure v1 and v2 with distinct physical data directories so deleting v2
// exercises both `old_dir` reservation and `is_surviving_data_dir`.
v1.data = None;
rustfs_utils::http::metadata_compat::remove_str(
&mut v1.metadata,
rustfs_utils::http::metadata_compat::SUFFIX_INLINE_DATA,
);
let v1_dir = Uuid::new_v4();
v1.data_dir = Some(v1_dir);
fs::create_dir_all(root.join(v1_dir.to_string()))
.await
.expect("create v1 dir");
fs::write(root.join(v1_dir.to_string()).join("part.1"), b"v1-data")
.await
.expect("write v1 part");
let mut v2 = v1.clone();
v2.version_id = Some(Uuid::new_v4());
let v2_dir = Uuid::new_v4();
v2.data_dir = Some(v2_dir);
fs::create_dir_all(root.join(v2_dir.to_string()))
.await
.expect("create v2 dir");
fs::write(root.join(v2_dir.to_string()).join("part.1"), b"v2-data")
.await
.expect("write v2 part");
let mut meta = FileMeta::new();
meta.add_version(v1.clone()).expect("add v1");
meta.add_version(v2.clone()).expect("add v2");
fs::write(root.join(STORAGE_FORMAT_FILE), meta.marshal_msg().expect("marshal meta"))
.await
.expect("write xl.meta");
let transaction = Uuid::new_v4();
let opts = DeleteOptions {
recoverable_delete: true,
..Default::default()
};
let fused = disk
.try_fused_recoverable_delete_version("recovery", "object", &v2, &opts, transaction, None)
.await
.expect("fused versioned delete preparation must succeed");
assert_eq!(fused, Some(()));
assert!(
root.join(v2_dir.to_string())
.join(format!("{RECOVERY_DELETE_DATA_DIR_MARKER_PREFIX}{transaction}"))
.is_file(),
"deleted version data_dir must have recovery marker"
);
let (journal, sole_transaction, target_satisfied) = disk
.try_publish_prepared_delete_local("recovery", "object", transaction, None)
.await
.expect("local publish must succeed")
.expect("versioned target must satisfy local publish");
assert!(sole_transaction, "surviving v1 data_dir must not clear sole_transaction");
assert!(target_satisfied);
assert!(journal.target_digest.is_some());
// Simulate interruption during finish-before-finalize right after TARGET is removed
// (after finished receipt + FINISHED marker were persisted, before receipt_path is removed).
let rollback = root.join(transaction.to_string());
let finished_receipt = CommitReceipt {
journal: journal.clone(),
finished: true,
};
fs::write(
rollback.join(DELETE_QUORUM_COMMITTED_FILE),
serde_json::to_vec(&finished_receipt).expect("serialize finished receipt"),
)
.await
.expect("write finished receipt");
fs::write(rollback.join(FINISHED), []).await.expect("write FINISHED");
fs::remove_file(rollback.join(STORAGE_FORMAT_FILE_BACKUP))
.await
.expect("remove backup");
fs::remove_file(rollback.join(TARGET)).await.expect("remove TARGET");
let inspected = disk
.inspect_delete_transaction("recovery", "object", transaction)
.await
.expect("interrupted post-TARGET state must remain inspectable without FileCorrupt");
assert_eq!(inspected.state, DeleteRecoveryState::CommitFinished);
// Complete cleanup via try_finish_and_finalize_prepared_delete_local.
let finalized = disk
.try_finish_and_finalize_prepared_delete_local("recovery", "object", transaction, &journal, None)
.await
.expect("local finalize must succeed after interruption");
assert!(finalized);
assert!(!rollback.exists(), "rollback directory must be pruned");
assert!(!root.join(v2_dir.to_string()).exists(), "deleted v2 data_dir must be removed");
assert!(
root.join(v1_dir.to_string()).join("part.1").is_file(),
"surviving v1 data_dir must remain intact"
);
}
#[tokio::test]
async fn delete_recovery_fused_unversioned_physical_delete_retains_receipt_until_finalize() {
let (_temp, disk, mut info, _) = fixture().await;
let root = disk.io_get_object_path("recovery", "object").expect("object path");
info.data = None;
rustfs_utils::http::metadata_compat::remove_str(
&mut info.metadata,
rustfs_utils::http::metadata_compat::SUFFIX_INLINE_DATA,
);
let data_dir = Uuid::new_v4();
info.data_dir = Some(data_dir);
fs::create_dir_all(root.join(data_dir.to_string()))
.await
.expect("create data_dir");
fs::write(root.join(data_dir.to_string()).join("part.1"), b"physical-payload")
.await
.expect("write part.1");
let mut meta = FileMeta::new();
meta.add_version(info.clone()).expect("add version");
fs::write(root.join(STORAGE_FORMAT_FILE), meta.marshal_msg().expect("marshal meta"))
.await
.expect("write xl.meta");
let transaction = Uuid::new_v4();
let opts = DeleteOptions {
recoverable_delete: true,
..Default::default()
};
let fused = disk
.try_fused_recoverable_delete_version("recovery", "object", &info, &opts, transaction, None)
.await
.expect("fused unversioned physical delete must succeed");
assert_eq!(fused, Some(()));
let (journal, sole_transaction, target_satisfied) = disk
.try_publish_prepared_delete_local("recovery", "object", transaction, None)
.await
.expect("local publish must succeed")
.expect("absent target must satisfy local publish");
assert!(sole_transaction);
assert!(target_satisfied);
assert!(journal.target_digest.is_none());
let rollback = root.join(transaction.to_string());
assert!(rollback.join(DELETE_QUORUM_COMMITTED_FILE).is_file());
let finalized = disk
.try_finish_and_finalize_prepared_delete_local("recovery", "object", transaction, &journal, None)
.await
.expect("local finalize must succeed");
assert!(finalized);
assert!(!rollback.exists(), "rollback directory must be pruned");
assert!(!root.join(data_dir.to_string()).exists(), "physical data_dir must be removed");
assert!(!root.exists(), "empty object directory must be pruned");
}
}
#[cfg(test)]
@@ -421,7 +421,6 @@ impl LocalDisk {
return Err(DiskError::ConditionalDeleteConflict);
}
let permit = AbortDeletePermit::new(&store, &expected, certificate.clone())?;
super::multipart_source_state::invalidate_active_multipart_source_head(&store.source);
let destination = self.io_get_object_path(RUSTFS_META_TMP_DELETED_BUCKET, &Uuid::new_v4().to_string())?;
let source = store.source.clone();
let base = store.base.clone();
@@ -619,9 +618,6 @@ pub(super) struct OrdinaryMultipartDeleteCheck {
root: os::PublicationRoot,
}
impl OrdinaryMultipartDeleteCheck {
pub(super) fn source(&self) -> &Path {
&self.source
}
pub(super) fn validate(&self) -> Result<()> {
let pinned = match os::pin_multipart_source_directory(&self.source, &self.base, &self.root) {
Ok(pinned) => pinned,
@@ -238,192 +238,9 @@ pub(super) fn read_optional_in_pinned(
Err(error) => Err(error.into()),
}
}
#[derive(Clone, Copy, PartialEq, Eq)]
struct FileStamp {
len: u64,
modified: Option<std::time::SystemTime>,
#[cfg(unix)]
ino: u64,
}
impl FileStamp {
fn from_path(path: &Path) -> Option<Self> {
let meta = std::fs::symlink_metadata(path).ok()?;
Some(Self {
len: meta.len(),
modified: meta.modified().ok(),
#[cfg(unix)]
ino: {
use std::os::unix::fs::MetadataExt;
meta.ino()
},
})
}
}
#[derive(Clone)]
struct ActiveMultipartSourceEntry {
raw: Bytes,
fi: FileInfo,
head_stamp: FileStamp,
unsettled_parts: std::collections::HashSet<usize>,
parts: std::collections::HashMap<usize, (FileStamp, rustfs_filemeta::ObjectPartInfo)>,
}
static ACTIVE_MULTIPART_SOURCE_HEADS: std::sync::LazyLock<
parking_lot::Mutex<std::collections::HashMap<PathBuf, ActiveMultipartSourceEntry>>,
> = std::sync::LazyLock::new(|| parking_lot::Mutex::new(std::collections::HashMap::new()));
pub(super) fn put_active_multipart_source_head(source: PathBuf, raw: Bytes, fi: FileInfo) {
let Some(head_stamp) = FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) else {
return;
};
let mut map = ACTIVE_MULTIPART_SOURCE_HEADS.lock();
if map.len() >= 4096
&& let Some(evict_key) = map.keys().next().cloned()
{
map.remove(&evict_key);
}
map.insert(
source,
ActiveMultipartSourceEntry {
raw,
fi,
head_stamp,
unsettled_parts: std::collections::HashSet::new(),
parts: std::collections::HashMap::new(),
},
);
}
fn stamp_matches_active_disk(source: &Path, expected_head_stamp: FileStamp) -> bool {
FileStamp::from_path(&source.join(STORAGE_FORMAT_FILE)) == Some(expected_head_stamp)
&& !source.join(STATE).exists()
&& !source.join(crate::disk::MULTIPART_RECOVERY_DIR).exists()
&& !source.join(".mpu-completions").exists()
}
pub(super) fn get_active_multipart_source_head(source: &Path) -> Option<(Bytes, FileInfo)> {
let (raw, fi, head_stamp) = {
let map = ACTIVE_MULTIPART_SOURCE_HEADS.lock();
let entry = map.get(source)?;
(entry.raw.clone(), entry.fi.clone(), entry.head_stamp)
};
if !stamp_matches_active_disk(source, head_stamp) {
ACTIVE_MULTIPART_SOURCE_HEADS.lock().remove(source);
return None;
}
Some((raw, fi))
}
pub(super) fn has_active_multipart_source_head(source: &Path) -> bool {
let Some(head_stamp) = ACTIVE_MULTIPART_SOURCE_HEADS.lock().get(source).map(|entry| entry.head_stamp) else {
return false;
};
if !stamp_matches_active_disk(source, head_stamp) {
ACTIVE_MULTIPART_SOURCE_HEADS.lock().remove(source);
return false;
}
true
}
pub(super) fn has_clean_active_multipart_source(source: &Path) -> bool {
let Some((head_stamp, clean)) = ACTIVE_MULTIPART_SOURCE_HEADS
.lock()
.get(source)
.map(|entry| (entry.head_stamp, entry.unsettled_parts.is_empty()))
else {
return false;
};
if !stamp_matches_active_disk(source, head_stamp) {
ACTIVE_MULTIPART_SOURCE_HEADS.lock().remove(source);
return false;
}
clean
}
pub(super) fn begin_active_multipart_part_transaction(source: &Path, part_number: usize) {
if let Some(entry) = ACTIVE_MULTIPART_SOURCE_HEADS.lock().get_mut(source) {
entry.unsettled_parts.insert(part_number);
entry.parts.remove(&part_number);
}
}
pub(super) fn record_active_multipart_part_committed(source: &Path, part: rustfs_filemeta::ObjectPartInfo) {
let Some(data_dir) = ACTIVE_MULTIPART_SOURCE_HEADS
.lock()
.get(source)
.and_then(|entry| entry.fi.data_dir)
.filter(|dir| !dir.is_nil())
else {
return;
};
let part_meta_path = source.join(data_dir.to_string()).join(format!("part.{}.meta", part.number));
let Some(part_stamp) = FileStamp::from_path(&part_meta_path) else {
return;
};
if let Some(entry) = ACTIVE_MULTIPART_SOURCE_HEADS.lock().get_mut(source) {
entry.parts.insert(part.number, (part_stamp, part));
}
}
pub(super) fn finish_active_multipart_part_transaction(
source: &Path,
part_number: Option<usize>,
committed: bool,
retired_existing_txn: bool,
) {
let mut map = ACTIVE_MULTIPART_SOURCE_HEADS.lock();
let Some(part_number) = part_number.filter(|_| committed) else {
map.remove(source);
return;
};
if !retired_existing_txn {
return;
}
if let Some(entry) = map.get_mut(source)
&& !entry.unsettled_parts.remove(&part_number)
{
map.remove(source);
}
}
pub(super) fn get_active_multipart_source_part(source: &Path, part_number: usize) -> Option<rustfs_filemeta::ObjectPartInfo> {
let (head_stamp, data_dir, expected_stamp, part) = {
let map = ACTIVE_MULTIPART_SOURCE_HEADS.lock();
let entry = map.get(source)?;
if !entry.unsettled_parts.is_empty() {
return None;
}
let data_dir = entry.fi.data_dir.filter(|dir| !dir.is_nil())?;
let (expected_stamp, part) = entry.parts.get(&part_number)?;
(entry.head_stamp, data_dir, *expected_stamp, part.clone())
};
if !stamp_matches_active_disk(source, head_stamp) {
ACTIVE_MULTIPART_SOURCE_HEADS.lock().remove(source);
return None;
}
let part_meta_path = source.join(data_dir.to_string()).join(format!("part.{part_number}.meta"));
if FileStamp::from_path(&part_meta_path) != Some(expected_stamp) {
return None;
}
Some(part)
}
pub(super) fn invalidate_active_multipart_source_head(source: &Path) {
ACTIVE_MULTIPART_SOURCE_HEADS.lock().remove(source);
}
pub(super) fn invalidate_active_multipart_sources_under(prefix: &Path) {
ACTIVE_MULTIPART_SOURCE_HEADS.lock().retain(|key, _| !key.starts_with(prefix));
}
/// Canonical source mutations are admitted under SourceRW at the syscall,
/// not on the earlier async caller that may have timed out.
pub(super) fn validate_source_mutation(source: &Path, base: &Path, root: &os::PublicationRoot, initial: bool) -> Result<()> {
if !initial && has_active_multipart_source_head(source) {
return Ok(());
}
let pinned = match os::pin_multipart_source_directory(source, base, root) {
Ok(pinned) => pinned,
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
@@ -623,7 +440,6 @@ impl SourceStore {
terminal: Option<&Path>,
sync_source_before: bool,
) -> std::io::Result<()> {
invalidate_active_multipart_source_head(source);
#[cfg(unix)]
let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
#[cfg(not(unix))]
@@ -654,7 +470,7 @@ impl SourceStore {
let terminal = terminal_record_path(&source, state);
let bytes = bytes.to_vec();
os::run_blocking_namespace_operation_with_sync(self.lease.clone(), true, move || {
Self::write_owned_backup_std(&source, &base, &root, owner, target, &bytes, false)?;
Self::write_owned_backup_std(&source, &base, &root, owner, target, &bytes)?;
Self::persist_state_std(&source, &base, &root, &raw, terminal.as_deref(), false)
})
.await
@@ -723,7 +539,7 @@ impl SourceStore {
let root = self.publication_root.clone();
let bytes = bytes.to_vec();
os::run_blocking_namespace_operation(self.lease.clone(), move || {
Self::write_owned_backup_std(&source, &base, &root, owner, target, &bytes, true)
Self::write_owned_backup_std(&source, &base, &root, owner, target, &bytes)
})
.await
.map_err(DiskError::from)
@@ -735,7 +551,6 @@ impl SourceStore {
owner: Uuid,
target: bool,
bytes: &[u8],
sync_source_root: bool,
) -> std::io::Result<()> {
let path = if target {
target_backup_path(source, owner)
@@ -767,10 +582,9 @@ impl SourceStore {
if !existed {
os::replace_multipart_source_metadata(&path, bytes, base, root)?;
}
os::sync_multipart_owned_record_parents_options(source, parent, existed, sync_source_root, base, root)
os::sync_multipart_owned_record_parents(source, parent, existed, base, root)
}
pub async fn replace_head(&self, bytes: &[u8]) -> Result<()> {
invalidate_active_multipart_source_head(&self.source);
let path = self.source.join(STORAGE_FORMAT_FILE);
let base = self.base.clone();
let root = self.publication_root.clone();
@@ -1283,82 +1097,6 @@ pub(super) fn load_source(
})
}
fn load_uninitialized_source(
raw: Bytes,
path: &str,
disk_id: Uuid,
physical_slot: usize,
disk_ids: &[Uuid],
) -> Result<LoadedSource> {
let sha = raw_sha(&raw);
let length = raw.len() as u64;
let current = get_file_info(
&raw,
RUSTFS_META_MULTIPART_BUCKET,
path,
"",
FileInfoOpts {
data: false,
include_free_versions: false,
include_part_checksums: true,
},
)
.map_err(Error::other)?;
native_source_protocol(&current)?;
if source_status(&current)?.is_some()
|| publication_identity(&current)?.is_some()
|| SourceConditionV1::take(&mut current.clone())?.is_some()
{
return Err(DiskError::FileCorrupt);
}
let bucket = rustfs_utils::http::metadata_compat::get_consistent_str(&current.metadata, "multipart-bucket")
.ok_or(DiskError::FileCorrupt)?;
let object = rustfs_utils::http::metadata_compat::get_consistent_str(&current.metadata, "multipart-object")
.ok_or(DiskError::FileCorrupt)?;
let condition = capture_source_condition(bucket, object, path, &current, current.data_dir.ok_or(DiskError::FileCorrupt)?)?;
if condition.generation.epoch != 0 {
return Err(DiskError::FileCorrupt);
}
let slot = condition
.lineage
.geometry
.distribution
.iter()
.position(|index| *index == current.erasure.index)
.ok_or(DiskError::FileCorrupt)?;
if slot != physical_slot || condition.lineage.geometry.disks() != disk_ids.len() {
return Err(DiskError::ConditionalDeleteConflict);
}
let inventory = raw_sha(&encode(&(None::<[u8; 32]>, Some(sha), sha, None::<[u8; 32]>, None::<[u8; 32]>))?);
let snapshot = Some(MultipartSourceSnapshot {
disk_id,
physical_slot: slot,
lineage: condition.lineage,
generation: condition.generation,
revision: 0,
phase: MultipartSourcePhase::Active,
state_sha256: [0; 32],
head_sha256: Some(sha),
inventory_sha256: inventory,
capability_sha256: None,
publication: None,
});
Ok(LoadedSource {
root_missing: false,
state: None,
head: Some(SourceHead {
sha256: sha,
length,
raw: Some(raw.clone()),
}),
original: Some(raw),
info: Some(current),
snapshot,
head_matches_state: true,
head_matches_activation: false,
})
}
pub(super) const RETIREMENT_ANCHOR: &str = ".mpu-source-retirement";
#[derive(Serialize, Deserialize)]
@@ -1585,6 +1323,41 @@ impl LocalDisk {
}
impl SourceStore {
/// Only native source validation may create provenance. Ordinary TMP/heal
/// publication can retain a valid old local anchor, never issue/rebind one.
pub async fn write_retirement_anchor(
&self,
state: &SourceState,
publication: &PublicationIdentity,
directory: &Path,
lease: Arc<os::NamespaceMutationLease>,
) -> Result<()> {
if state.lineage != publication.lineage
|| state.generation != publication.source_generation
|| state.disk_id != self.disk_id
|| state.physical_slot != self.physical_slot
{
return Err(DiskError::ConditionalDeleteConflict);
}
let record = NativeRetirementAnchor {
protocol_version: MULTIPART_SOURCE_PROTOCOL_VERSION,
disk_id: self.disk_id,
physical_slot: self.physical_slot,
publication: publication.clone(),
source_original_sha256: state.original_sha256,
source_validation_receipt_id: state.backup_owner,
};
let bytes = encode(&record)?;
let directory = directory.to_path_buf();
let base = self.base.clone();
let root = self.publication_root.clone();
os::run_blocking_namespace_operation(lease, move || {
let _parent = os::mkdir_all_below_existing_base_std(&directory, &base, &root)?;
os::replace_multipart_source_metadata(&directory.join(RETIREMENT_ANCHOR), &bytes, &base, &root)
})
.await
.map_err(DiskError::from)
}
/// This caller already owns SourceRW and destination metadata ownership.
/// An Aborting survivor can confirm a historical target only with its
/// independent, same-disk publication anchor. Closure never creates one.
@@ -1683,8 +1456,6 @@ struct ValidatedPreparedPublication {
length: u64,
}
type DeferredInitialPublication = (Vec<u8>, PathBuf, Vec<u8>);
pub(super) struct NativeMultipartPublication {
pub store: Arc<SourceStore>,
pub publication: PublicationIdentity,
@@ -1692,49 +1463,22 @@ pub(super) struct NativeMultipartPublication {
state: parking_lot::Mutex<SourceState>,
capability: parking_lot::Mutex<Option<MultipartUndoCapability>>,
validated_prepared: parking_lot::Mutex<Option<ValidatedPreparedPublication>>,
deferred_initial: parking_lot::Mutex<Option<DeferredInitialPublication>>,
pub legacy: bool,
}
impl LocalDisk {
#[cfg(test)]
pub(super) async fn prepare_native_multipart_publication(
&self,
path: &str,
incoming: &mut FileInfo,
staging: &MultipartCommitStaging,
) -> Result<NativeMultipartPublication> {
self.prepare_native_multipart_publication_inner(path, incoming, staging, false)
.await
}
pub(super) async fn prepare_native_multipart_publication_deferred(
&self,
path: &str,
incoming: &mut FileInfo,
staging: &MultipartCommitStaging,
) -> Result<NativeMultipartPublication> {
self.prepare_native_multipart_publication_inner(path, incoming, staging, true)
.await
}
async fn prepare_native_multipart_publication_inner(
&self,
path: &str,
incoming: &mut FileInfo,
staging: &MultipartCommitStaging,
defer_persist: bool,
) -> Result<NativeMultipartPublication> {
// Real Complete is parts-backed, including a zero-byte last part.
if incoming.data.is_some() || incoming.inline_data() {
return Err(DiskError::UnsupportedDisk);
}
let store = Arc::new(self.multipart_store_from_staging(path, staging).await?);
let loaded = if defer_persist && let Some(raw) = staging.fresh_source_head() {
load_uninitialized_source(raw, &store.path, store.disk_id, store.physical_slot, &store.disk_ids)?
} else {
store.load().await?
};
let loaded = store.load().await?;
let snapshot = loaded.snapshot.as_ref().ok_or(DiskError::ConditionalDeleteConflict)?;
let original = loaded
.head
@@ -1825,44 +1569,7 @@ impl LocalDisk {
};
// Copy the small source FI to an independent immutable inode. Its
// canonical inode is never truncated to build the destination proxy.
let directory = staging.private_directory(publication.final_data_dir);
let anchor_record = NativeRetirementAnchor {
protocol_version: MULTIPART_SOURCE_PROTOCOL_VERSION,
disk_id: store.disk_id,
physical_slot: store.physical_slot,
publication: publication.clone(),
source_original_sha256: state.original_sha256,
source_validation_receipt_id: state.backup_owner,
};
let anchor_bytes = encode(&anchor_record)?;
state.validate(store.disk_id, &store.path)?;
let original_bytes = original.to_vec();
let deferred_initial = if defer_persist {
Some((original_bytes, directory, anchor_bytes))
} else {
let state_raw = encode(&state)?;
let source_dir = store.source.clone();
let base_dir = store.base.clone();
let pub_root = store.publication_root.clone();
let terminal = terminal_record_path(&source_dir, &state);
#[cfg(target_os = "linux")]
let can_mark_presynced = has_active_multipart_source_head(&source_dir)
&& super::effective_durability(RUSTFS_META_MULTIPART_BUCKET).syncs_data_shards();
os::run_blocking_namespace_operation_with_sync(staging.lease.clone(), true, move || {
SourceStore::write_owned_backup_std(&source_dir, &base_dir, &pub_root, owner, false, &original_bytes, false)?;
SourceStore::persist_state_std(&source_dir, &base_dir, &pub_root, &state_raw, terminal.as_deref(), false)?;
let _parent = os::mkdir_all_below_existing_base_std(&directory, &base_dir, &pub_root)?;
os::replace_multipart_source_metadata(&directory.join(RETIREMENT_ANCHOR), &anchor_bytes, &base_dir, &pub_root)?;
#[cfg(target_os = "linux")]
if can_mark_presynced {
super::mark_presynced_tmp_data_dir(directory);
}
Ok(())
})
.await
.map_err(DiskError::from)?;
None
};
store.write_owned_backup_and_persist(owner, false, original, &state).await?;
let prepared_path = store.source.join(format!(".mpu-prepared-{owner}"));
let context = NativeMultipartPublication {
store,
@@ -1871,9 +1578,14 @@ impl LocalDisk {
state: parking_lot::Mutex::new(state),
capability: parking_lot::Mutex::new(Some(capability)),
validated_prepared: parking_lot::Mutex::new(None),
deferred_initial: parking_lot::Mutex::new(deferred_initial),
legacy,
};
let directory = staging.private_directory(context.publication.final_data_dir);
let reserved = context.state.lock().clone();
context
.store
.write_retirement_anchor(&reserved, &context.publication, &directory, staging.lease.clone())
.await?;
Ok(context)
}
}
@@ -1882,15 +1594,18 @@ impl NativeMultipartPublication {
pub fn canonical_head(&self) -> PathBuf {
self.store.source.join(STORAGE_FORMAT_FILE)
}
#[cfg(test)]
pub async fn prepare_metadata(&self, new_metadata: &[u8], old_metadata: Option<&[u8]>) -> Result<()> {
let mut state = self.state.lock().clone();
let current = self.store.load().await?;
let current_state = current.state.as_ref().ok_or(DiskError::ConditionalDeleteConflict)?;
if encode(current_state)? != encode(&state)? || !current.head_matches_state {
return Err(DiskError::ConditionalDeleteConflict);
}
// Validate the actual persisted representation once, after FileMeta's
// canonicalization. No unvalidated input can install a source proxy.
if !self.store.target_matches(&self.publication, new_metadata)? {
return Err(DiskError::ConditionalDeleteConflict);
}
let expected_state_raw = encode(&state)?;
let validated = ValidatedPreparedPublication {
publication: self.publication.clone(),
sha256: raw_sha(new_metadata),
@@ -1903,297 +1618,17 @@ impl NativeMultipartPublication {
state.prepared_sha256 = Some(validated.sha256);
state.prepared_length = Some(validated.length);
state.advance(MultipartSourcePhase::Prepared)?;
state.validate(self.store.disk_id, &self.store.path)?;
let next_state_raw = encode(&state)?;
let source = self.store.source.clone();
let base = self.store.base.clone();
let root = self.store.publication_root.clone();
let path = self.store.path.clone();
let disk_id = self.store.disk_id;
let physical_slot = self.store.physical_slot;
let disk_ids = self.store.disk_ids.clone();
let backup_owner = state.backup_owner;
let old_bytes = old_metadata.map(|b| b.to_vec());
let terminal = terminal_record_path(&source, &state);
os::run_blocking_namespace_operation_with_sync(self.store.lease.clone(), true, move || {
let current = load_source(&source, &base, &root, &path, disk_id, physical_slot, &disk_ids).map_err(IoError::other)?;
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current_raw != expected_state_raw || !current.head_matches_state {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
if let Some(previous) = old_bytes.as_deref() {
SourceStore::write_owned_backup_std(&source, &base, &root, backup_owner, true, previous, false)?;
}
SourceStore::persist_state_std(&source, &base, &root, &next_state_raw, terminal.as_deref(), false)
})
.await
.map_err(DiskError::from)?;
if let Some(previous) = old_metadata {
self.store
.write_owned_backup_and_persist(state.backup_owner, true, previous, &state)
.await?;
} else {
self.store.persist(&state).await?;
}
*self.state.lock() = state;
*self.validated_prepared.lock() = Some(validated);
Ok(())
}
pub(super) fn prepare_and_install_head_on_thread(
&self,
new_metadata: &[u8],
old_metadata: Option<&[u8]>,
dst_file_path: &Path,
directory_guard: &os::RenameCommitGuard,
) -> Result<os::PreparedRenameSource> {
let mut state = self.state.lock().clone();
if !self.store.target_matches(&self.publication, new_metadata)? {
return Err(DiskError::ConditionalDeleteConflict);
}
let deferred_initial = self.deferred_initial.lock().take();
let expected_state_raw = encode(&state)?;
let expected_lineage = state.lineage.clone();
let expected_generation = state.generation;
let expected_revision = state.revision;
let validated = ValidatedPreparedPublication {
publication: self.publication.clone(),
sha256: raw_sha(new_metadata),
length: new_metadata.len() as u64,
};
if let Some(previous) = old_metadata {
state.previous_target_sha256 = Some(raw_sha(previous));
state.previous_target_length = Some(previous.len() as u64);
}
state.prepared_sha256 = Some(validated.sha256);
state.prepared_length = Some(validated.length);
state.advance(MultipartSourcePhase::Prepared)?;
state.validate(self.store.disk_id, &self.store.path)?;
let next_state_raw = encode(&state)?;
let source = &self.store.source;
let base = &self.store.base;
let root = &self.store.publication_root;
let path = &self.store.path;
let disk_id = self.store.disk_id;
let physical_slot = self.store.physical_slot;
let disk_ids = &self.store.disk_ids;
let backup_owner = state.backup_owner;
#[cfg(target_os = "linux")]
let can_mark_presynced = has_active_multipart_source_head(source)
&& super::effective_durability(RUSTFS_META_MULTIPART_BUCKET).syncs_data_shards();
let terminal = terminal_record_path(source, &state);
let prepared_path = &self.prepared_path;
let canonical = self.canonical_head();
let prepared = (|| -> std::io::Result<os::PreparedRenameSource> {
if let Some((ref original_bytes, _, _)) = deferred_initial {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let snapshot = current
.snapshot
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_head = current
.head
.as_ref()
.and_then(SourceHead::raw)
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
if !current.head_matches_state
|| snapshot.phase != MultipartSourcePhase::Active
|| snapshot.capability_sha256.is_some()
|| snapshot.lineage != expected_lineage
|| snapshot.generation != expected_generation
|| snapshot.revision.checked_add(1) != Some(expected_revision)
|| current_head != original_bytes.as_slice()
{
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
} else {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current_raw != expected_state_raw || !current.head_matches_state {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
}
#[cfg(test)]
super::run_owned_file_before_open(prepared_path);
let mut prepared_metadata_source =
os::create_prepared_rename_source_with_commit_guard(prepared_path, dst_file_path, directory_guard)?;
if let Err(err) = prepared_metadata_source.write_all(new_metadata, false) {
drop(prepared_metadata_source);
let _ = std::fs::remove_file(prepared_path);
return Err(err);
}
let setup_res: std::io::Result<()> = (|| {
if let Some((original_bytes, directory, anchor_bytes)) = deferred_initial {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, false, &original_bytes, false)?;
if let Some(previous) = old_metadata {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, true, previous, false)?;
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), false)?;
let _parent = os::mkdir_all_below_existing_base_std(&directory, base, root)?;
os::replace_multipart_source_metadata(&directory.join(RETIREMENT_ANCHOR), &anchor_bytes, base, root)?;
#[cfg(target_os = "linux")]
if can_mark_presynced {
super::mark_presynced_tmp_data_dir(directory);
}
} else {
if let Some(previous) = old_metadata {
SourceStore::write_owned_backup_std(source, base, root, backup_owner, true, previous, false)?;
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), false)?;
}
Ok(())
})();
if let Err(err) = setup_res {
drop(prepared_metadata_source);
let _ = std::fs::remove_file(prepared_path);
return Err(err);
}
if let Err(err) = prepared_metadata_source.sync_data() {
drop(prepared_metadata_source);
let _ = std::fs::remove_file(prepared_path);
return Err(err);
}
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, base, root)
})()
.map_err(DiskError::from)?;
*self.state.lock() = state;
*self.validated_prepared.lock() = Some(validated);
Ok(prepared)
}
#[cfg(not(target_os = "linux"))]
pub async fn prepare_and_install_head(
&self,
new_metadata: &[u8],
old_metadata: Option<&[u8]>,
dst_file_path: &Path,
directory_guard: &os::RenameCommitGuard,
mutation_lease: Arc<os::NamespaceMutationLease>,
) -> Result<os::PreparedRenameSource> {
let mut state = self.state.lock().clone();
if !self.store.target_matches(&self.publication, new_metadata)? {
return Err(DiskError::ConditionalDeleteConflict);
}
let deferred_initial = self.deferred_initial.lock().take();
let expected_state_raw = encode(&state)?;
let expected_lineage = state.lineage.clone();
let expected_generation = state.generation;
let expected_revision = state.revision;
let validated = ValidatedPreparedPublication {
publication: self.publication.clone(),
sha256: raw_sha(new_metadata),
length: new_metadata.len() as u64,
};
if let Some(previous) = old_metadata {
state.previous_target_sha256 = Some(raw_sha(previous));
state.previous_target_length = Some(previous.len() as u64);
}
state.prepared_sha256 = Some(validated.sha256);
state.prepared_length = Some(validated.length);
state.advance(MultipartSourcePhase::Prepared)?;
state.validate(self.store.disk_id, &self.store.path)?;
let next_state_raw = encode(&state)?;
let source = self.store.source.clone();
let base = self.store.base.clone();
let root = self.store.publication_root.clone();
let path = self.store.path.clone();
let disk_id = self.store.disk_id;
let physical_slot = self.store.physical_slot;
let disk_ids = self.store.disk_ids.clone();
let backup_owner = state.backup_owner;
#[cfg(target_os = "linux")]
let can_mark_presynced = has_active_multipart_source_head(&source)
&& super::effective_durability(RUSTFS_META_MULTIPART_BUCKET).syncs_data_shards();
let old_bytes = old_metadata.map(|b| b.to_vec());
let new_bytes = new_metadata.to_vec();
let terminal = terminal_record_path(&source, &state);
let prepared_path = self.prepared_path.clone();
let canonical = self.canonical_head();
let dst_file_path = dst_file_path.to_path_buf();
let owned_directory_guard = directory_guard.clone();
let prepared = os::run_blocking_namespace_operation_with_sync(mutation_lease, true, move || {
let current = load_source(&source, &base, &root, &path, disk_id, physical_slot, &disk_ids).map_err(IoError::other)?;
if let Some((original_bytes, directory, anchor_bytes)) = deferred_initial {
let snapshot = current
.snapshot
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_head = current
.head
.as_ref()
.and_then(SourceHead::raw)
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
if !current.head_matches_state
|| snapshot.phase != MultipartSourcePhase::Active
|| snapshot.capability_sha256.is_some()
|| snapshot.lineage != expected_lineage
|| snapshot.generation != expected_generation
|| snapshot.revision.checked_add(1) != Some(expected_revision)
|| current_head != original_bytes.as_slice()
{
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
SourceStore::write_owned_backup_std(&source, &base, &root, backup_owner, false, &original_bytes, false)?;
if let Some(previous) = old_bytes.as_deref() {
SourceStore::write_owned_backup_std(&source, &base, &root, backup_owner, true, previous, false)?;
}
SourceStore::persist_state_std(&source, &base, &root, &next_state_raw, terminal.as_deref(), false)?;
let _parent = os::mkdir_all_below_existing_base_std(&directory, &base, &root)?;
os::replace_multipart_source_metadata(&directory.join(RETIREMENT_ANCHOR), &anchor_bytes, &base, &root)?;
#[cfg(target_os = "linux")]
if can_mark_presynced {
super::mark_presynced_tmp_data_dir(directory);
}
} else {
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current_raw != expected_state_raw || !current.head_matches_state {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
if let Some(previous) = old_bytes.as_deref() {
SourceStore::write_owned_backup_std(&source, &base, &root, backup_owner, true, previous, false)?;
}
SourceStore::persist_state_std(&source, &base, &root, &next_state_raw, terminal.as_deref(), false)?;
}
#[cfg(test)]
super::run_owned_file_before_open(&prepared_path);
let mut prepared_metadata_source =
os::create_prepared_rename_source_with_commit_guard(&prepared_path, &dst_file_path, &owned_directory_guard)?;
prepared_metadata_source.write_all(&new_bytes, true)?;
os::install_multipart_prepared_head(prepared_metadata_source, &canonical, &base, &root)
})
.await
.map_err(DiskError::from)?;
*self.state.lock() = state;
*self.validated_prepared.lock() = Some(validated);
Ok(prepared)
}
#[cfg(target_os = "linux")]
pub async fn prepare_and_install_head(
self: &Arc<Self>,
new_metadata: &[u8],
old_metadata: Option<&[u8]>,
dst_file_path: &Path,
directory_guard: &os::RenameCommitGuard,
mutation_lease: Arc<os::NamespaceMutationLease>,
) -> Result<os::PreparedRenameSource> {
let this = Arc::clone(self);
let new_bytes = new_metadata.to_vec();
let old_bytes = old_metadata.map(|b| b.to_vec());
let dst_file_path = dst_file_path.to_path_buf();
let owned_directory_guard = directory_guard.clone();
os::run_blocking_namespace_operation_with_sync(mutation_lease, true, move || {
this.prepare_and_install_head_on_thread(&new_bytes, old_bytes.as_deref(), &dst_file_path, &owned_directory_guard)
.map_err(IoError::other)
})
.await
.map_err(DiskError::from)
}
#[cfg(test)]
pub async fn install_head(&self, prepared: os::PreparedRenameSource) -> Result<os::PreparedRenameSource> {
{
let guard = self.validated_prepared.lock();
@@ -2264,13 +1699,13 @@ impl NativeMultipartPublication {
// return an error after consuming the last claimable source heads.
self.store.restore_exact_target_with_lease(&state, lease).await
}
pub(super) fn finish_on_thread(
&self,
target_already_synced: bool,
staging_cleanup: Option<(PathBuf, Arc<std::sync::atomic::AtomicBool>)>,
) -> Result<MultipartPublicationResult> {
pub async fn finish(&self, target_lease: Arc<os::NamespaceMutationLease>) -> Result<MultipartPublicationResult> {
let mut state = self.state.lock().clone();
let expected_state_raw = encode(&state)?;
let current = self.store.load().await?;
let current_state = current.state.as_ref().ok_or(DiskError::ConditionalDeleteConflict)?;
if current.head.is_some() || encode(current_state)? != encode(&state)? {
return Err(DiskError::ConditionalDeleteConflict);
}
let (expected_sha256, expected_length) = {
let guard = self.validated_prepared.lock();
let validated = guard.as_ref().ok_or(DiskError::ConditionalDeleteConflict)?;
@@ -2282,88 +1717,16 @@ impl NativeMultipartPublication {
}
(validated.sha256, validated.length)
};
let target_dir = self.store.target_path(&self.publication)?;
self.store
.read_and_sync_target_with_lease(&self.publication, target_lease, move |target| {
if target.len() as u64 != expected_length || raw_sha(target) != expected_sha256 {
return Err(DiskError::ConditionalDeleteConflict);
}
Ok(())
})
.await?;
state.advance(MultipartSourcePhase::ConsumedUnsealed)?;
state.validate(self.store.disk_id, &self.store.path)?;
let next_state_raw = encode(&state)?;
let snapshot = snapshot_for_state(self.store.disk_id, &state, &next_state_raw, None)?;
let source = &self.store.source;
let base = &self.store.base;
let root = &self.store.publication_root;
let path = &self.store.path;
let disk_id = self.store.disk_id;
let physical_slot = self.store.physical_slot;
let disk_ids = &self.store.disk_ids;
let terminal = terminal_record_path(source, &state);
(|| -> std::io::Result<()> {
let current = load_source(source, base, root, path, disk_id, physical_slot, disk_ids).map_err(IoError::other)?;
let current_state = current
.state
.as_ref()
.ok_or_else(|| IoError::other(DiskError::ConditionalDeleteConflict))?;
let current_raw = encode(current_state).map_err(IoError::other)?;
if current.head.is_some() || current_raw != expected_state_raw {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
if target_already_synced {
let pinned = os::pin_multipart_source_directory(&target_dir, base, root).map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
let target = os::read_multipart_source_metadata_in_pinned(
&pinned,
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
if target.len() as u64 != expected_length || raw_sha(&target) != expected_sha256 {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
} else {
os::read_and_sync_multipart_target_metadata(
&target_dir,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
base,
root,
|target| {
if target.len() as u64 != expected_length || raw_sha(target) != expected_sha256 {
return Err(IoError::other(DiskError::ConditionalDeleteConflict));
}
Ok(())
},
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})?;
}
SourceStore::persist_state_std(source, base, root, &next_state_raw, terminal.as_deref(), true)?;
if let Some((io_path, cleaned)) = staging_cleanup {
match std::fs::remove_dir(&io_path) {
Ok(()) => cleaned.store(true, std::sync::atomic::Ordering::Release),
Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
cleaned.store(true, std::sync::atomic::Ordering::Release);
}
Err(_) => {}
}
}
Ok(())
})()
.map_err(DiskError::from)?;
let snapshot = self.store.persist_with_source_sync(&state, true).await?;
*self.state.lock() = state;
let proof = self
.store
@@ -2375,21 +1738,6 @@ impl NativeMultipartPublication {
undo_capability: capability,
})
}
pub async fn finish(
self: &Arc<Self>,
target_lease: Arc<os::NamespaceMutationLease>,
target_already_synced: bool,
staging_cleanup: Option<(PathBuf, Arc<std::sync::atomic::AtomicBool>)>,
) -> Result<MultipartPublicationResult> {
let this = Arc::clone(self);
os::run_blocking_namespace_operation_with_sync(target_lease, true, move || {
this.finish_on_thread(target_already_synced, staging_cleanup)
.map_err(IoError::other)
})
.await
.map_err(DiskError::from)
}
}
impl SourceStore {
@@ -207,12 +207,6 @@ pub(super) struct MultipartSourceAdmission {
initial: bool,
}
impl MultipartSourceAdmission {
pub(super) fn source(&self) -> &Path {
&self.source
}
pub(super) fn is_initial(&self) -> bool {
self.initial
}
pub(super) fn precondition(&self) -> Arc<dyn Fn() -> std::io::Result<()> + Send + Sync> {
let admission = self.clone();
Arc::new(move || admission.validate())
@@ -256,8 +250,6 @@ pub(super) struct MultipartCommitStaging {
logical_path: PathBuf,
io_path: PathBuf,
owner: Option<Arc<dyn Send + Sync>>,
cleaned: Arc<std::sync::atomic::AtomicBool>,
fresh_source_head: Option<bytes::Bytes>,
}
impl MultipartCommitStaging {
@@ -267,19 +259,10 @@ impl MultipartCommitStaging {
pub(super) fn private_directory(&self, directory: Uuid) -> PathBuf {
self.io_path.join(directory.to_string())
}
pub(super) fn cleanup_handle(&self) -> (PathBuf, Arc<std::sync::atomic::AtomicBool>) {
(self.io_path.clone(), Arc::clone(&self.cleaned))
}
pub(super) fn fresh_source_head(&self) -> Option<bytes::Bytes> {
self.fresh_source_head.clone()
}
}
impl Drop for MultipartCommitStaging {
fn drop(&mut self) {
if self.cleaned.load(std::sync::atomic::Ordering::Acquire) {
return;
}
let logical_path = self.logical_path.clone();
let io_path = self.io_path.clone();
let owner = self.owner.clone();
@@ -302,14 +285,6 @@ impl Drop for MultipartCommitStaging {
}
impl LocalDisk {
pub(crate) fn has_clean_active_multipart_source(&self, upload_or_part_path: &str, part_path: bool) -> bool {
let root = upload_scope_root(upload_or_part_path, part_path);
let Ok(source) = self.io_get_object_path(RUSTFS_META_MULTIPART_BUCKET, root) else {
return false;
};
super::multipart_source_state::has_clean_active_multipart_source(&source)
}
pub(super) fn multipart_source_admission(
&self,
volume: &str,
@@ -381,14 +356,12 @@ impl LocalDisk {
let io_path = self.io_get_object_path(RUSTFS_META_TMP_BUCKET, &attempt)?;
// The attempt root is exactly the namespace used by the tmp sweeper.
let lease = os::acquire_namespace_mutation_lease_with_owner(&logical_path, owner.clone()).await;
let mut staging = MultipartCommitStaging {
let staging = MultipartCommitStaging {
path: attempt,
lease,
logical_path,
io_path,
owner,
cleaned: Arc::new(std::sync::atomic::AtomicBool::new(false)),
fresh_source_head: None,
};
let canonical_metadata = self.io_get_object_path(RUSTFS_META_MULTIPART_BUCKET, &format!("{source_path}/xl.meta"))?;
let source_object = self.io_get_object_path(RUSTFS_META_MULTIPART_BUCKET, source_path)?;
@@ -398,37 +371,31 @@ impl LocalDisk {
let source_path = source_path.to_owned();
let incoming = incoming.clone();
let sync_directory = effective_durability(destination_volume).syncs_data_shards();
let fresh_head = os::run_cancellable_blocking_namespace_operation(Arc::clone(&staging.lease), move |cancelled| {
os::run_cancellable_blocking_namespace_operation(Arc::clone(&staging.lease), move |cancelled| {
if cancelled() {
return Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "multipart staging cancelled"));
}
let cached_head = super::multipart_source_state::get_active_multipart_source_head(&source_object);
let (raw, source, from_active_cache) = if let Some((raw, source)) = cached_head {
(raw, source, true)
} else {
let raw = os::read_multipart_source_metadata(&canonical_metadata, &base, &publication_root).map_err(|error| {
// A retired source has no authority for this delayed completion.
// Preserve confinement and other I/O failures as their real errors.
if error.kind() == std::io::ErrorKind::NotFound {
DiskError::FileNotFound.into()
} else {
error
}
})?;
let source = get_file_info(
&raw,
RUSTFS_META_MULTIPART_BUCKET,
&source_path,
"",
FileInfoOpts {
data: false,
include_free_versions: false,
include_part_checksums: true,
},
)
.map_err(std::io::Error::other)?;
(bytes::Bytes::from(raw), source, false)
};
let raw = os::read_multipart_source_metadata(&canonical_metadata, &base, &publication_root).map_err(|error| {
// A retired source has no authority for this delayed completion.
// Preserve confinement and other I/O failures as their real errors.
if error.kind() == std::io::ErrorKind::NotFound {
DiskError::FileNotFound.into()
} else {
error
}
})?;
let source = get_file_info(
&raw,
RUSTFS_META_MULTIPART_BUCKET,
&source_path,
"",
FileInfoOpts {
data: false,
include_free_versions: false,
include_part_checksums: true,
},
)
.map_err(std::io::Error::other)?;
let source_dir = source
.data_dir
.filter(|dir| !dir.is_nil())
@@ -445,31 +412,23 @@ impl LocalDisk {
let _attempt = os::mkdir_all_below_existing_base_std(&private_object, &base, &publication_root)?;
if incoming.data.is_none() && !incoming.is_remote() {
let source_data_dir = source_object.join(source_dir.to_string());
let mut pinned_source_data = None;
let pinned_source_data = (!incoming.parts.is_empty())
.then(|| os::pin_multipart_source_directory(&source_data_dir, &base, &publication_root))
.transpose()?;
let mut names = Vec::with_capacity(incoming.parts.len().saturating_mul(3));
for part in &incoming.parts {
if cancelled() {
return Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "multipart staging cancelled"));
}
let meta_name = format!("part.{}.meta", part.number);
let actual = if let Some(cached_part) =
super::multipart_source_state::get_active_multipart_source_part(&source_object, part.number)
{
cached_part
} else {
if pinned_source_data.is_none() {
pinned_source_data =
Some(os::pin_multipart_source_directory(&source_data_dir, &base, &publication_root)?);
}
let bytes = os::read_multipart_source_metadata_in_pinned(
pinned_source_data.as_ref().expect("pinned source data directory"),
&source_data_dir,
std::ffi::OsStr::new(&meta_name),
&base,
&publication_root,
)?;
rustfs_filemeta::ObjectPartInfo::unmarshal(&bytes).map_err(std::io::Error::other)?
};
let bytes = os::read_multipart_source_metadata_in_pinned(
pinned_source_data.as_ref().expect("pinned source data directory"),
&source_data_dir,
std::ffi::OsStr::new(&meta_name),
&base,
&publication_root,
)?;
let actual = rustfs_filemeta::ObjectPartInfo::unmarshal(&bytes).map_err(std::io::Error::other)?;
let normalized = i64::try_from(actual.size)
.map_err(std::io::Error::other)?
.max(actual.actual_size);
@@ -505,15 +464,10 @@ impl LocalDisk {
cancelled,
)?;
}
let has_state = !from_active_cache
&& (source_object.join(".mpu-source-state").exists()
|| source_object.join(".mpu-recovery").exists()
|| source_object.join(".mpu-completions").exists());
Ok((!has_state).then_some(raw))
Ok(())
})
.await
.map_err(DiskError::from)?;
staging.fresh_source_head = fresh_head;
Ok(staging)
}
}
@@ -1265,211 +1219,4 @@ mod tests {
.expect("non-part empty hash cleanup must remain supported");
assert!(!directory.exists(), "a per-upload safety guard must not strand empty hash directories");
}
#[tokio::test]
async fn multipart_source_active_head_cache_populates_and_invalidates_across_part_and_delete_paths() {
use crate::disk::{DiskAPI as _, PartTransactionAction, ReadOptions};
let (_root, disk, completed, _) = fixture().await;
std::fs::create_dir_all(disk.io_get_bucket_path(RUSTFS_META_TMP_BUCKET).expect("TMP path")).expect("TMP bucket");
let upload = format!("{}/{}", "c".repeat(64), Uuid::new_v4());
let data_dir = Uuid::new_v4();
let mut init_fi = FileInfo::new(&upload, 2, 2);
init_fi.erasure.index = 1;
init_fi.data_dir = Some(data_dir);
init_fi.fresh = true;
init_fi.mod_time = Some(time::OffsetDateTime::now_utc());
disk.write_metadata("", RUSTFS_META_MULTIPART_BUCKET, &upload, init_fi.clone())
.await
.expect("fresh multipart upload metadata");
assert!(disk.has_clean_active_multipart_source(&upload, false));
let read_fi = disk
.read_version("", RUSTFS_META_MULTIPART_BUCKET, &upload, "", &ReadOptions::default())
.await
.expect("cached read_version");
assert_eq!(read_fi.data_dir, Some(data_dir));
let destination = format!("{upload}/{data_dir}/part.1");
let part_meta = bytes::Bytes::from(completed.parts[0].marshal_msg().expect("part meta"));
disk.write_all(
RUSTFS_META_TMP_BUCKET,
"cache-test-part/part.1",
bytes::Bytes::from_static(b"immutable shard"),
)
.await
.expect("stage tmp part");
disk.prepare_part_transaction(
RUSTFS_META_TMP_BUCKET,
"cache-test-part/part.1",
RUSTFS_META_MULTIPART_BUCKET,
&destination,
part_meta.clone(),
)
.await
.expect("prepare part transaction");
assert!(
!disk.has_clean_active_multipart_source(&upload, false),
"in-flight part transaction must mark active source as having unsettled part txn"
);
disk.rename_part(
RUSTFS_META_TMP_BUCKET,
"cache-test-part/part.1",
RUSTFS_META_MULTIPART_BUCKET,
&destination,
part_meta,
)
.await
.expect("rename part");
disk.settle_part_transaction(RUSTFS_META_MULTIPART_BUCKET, &destination, PartTransactionAction::Commit)
.await
.expect("settle part transaction");
assert!(
disk.has_clean_active_multipart_source(&upload, false),
"settled commit must restore clean active multipart source status"
);
let read_parts = disk
.read_parts(RUSTFS_META_MULTIPART_BUCKET, &[format!("{destination}.meta")])
.await
.expect("cached read_parts");
assert_eq!(read_parts.len(), 1);
assert_eq!(read_parts[0].number, 1);
// Prepare part 2, then replay Commit for already-settled part 1: part 2's
// pending transaction must keep the upload marked as having unsettled part txns.
let destination2 = format!("{upload}/{data_dir}/part.2");
let mut part2_info = completed.parts[0].clone();
part2_info.number = 2;
let part2_meta = bytes::Bytes::from(part2_info.marshal_msg().expect("part 2 meta"));
disk.write_all(
RUSTFS_META_TMP_BUCKET,
"cache-test-part/part.2",
bytes::Bytes::from_static(b"immutable shard"),
)
.await
.expect("stage tmp part 2");
disk.prepare_part_transaction(
RUSTFS_META_TMP_BUCKET,
"cache-test-part/part.2",
RUSTFS_META_MULTIPART_BUCKET,
&destination2,
part2_meta.clone(),
)
.await
.expect("prepare part 2 transaction");
assert!(!disk.has_clean_active_multipart_source(&upload, false));
disk.settle_part_transaction(RUSTFS_META_MULTIPART_BUCKET, &destination, PartTransactionAction::Commit)
.await
.expect("replayed part 1 commit settlement");
assert!(
!disk.has_clean_active_multipart_source(&upload, false),
"replayed settlement for part 1 must not clear part 2's pending transaction"
);
disk.rename_part(
RUSTFS_META_TMP_BUCKET,
"cache-test-part/part.2",
RUSTFS_META_MULTIPART_BUCKET,
&destination2,
part2_meta,
)
.await
.expect("rename part 2");
disk.settle_part_transaction(RUSTFS_META_MULTIPART_BUCKET, &destination2, PartTransactionAction::Commit)
.await
.expect("settle part 2 transaction");
assert!(disk.has_clean_active_multipart_source(&upload, false));
// Deleting {upload}/xl.meta through delete_paths must invalidate the canonical {upload} cache key.
disk.delete_paths(RUSTFS_META_MULTIPART_BUCKET, &[format!("{upload}/xl.meta")])
.await
.expect("delete xl.meta via delete_paths");
assert!(
!disk.has_clean_active_multipart_source(&upload, false),
"per-file head deletion must invalidate the canonical upload cache key"
);
assert!(matches!(
disk.read_version("", RUSTFS_META_MULTIPART_BUCKET, &upload, "", &ReadOptions::default())
.await,
Err(DiskError::FileNotFound)
));
}
#[cfg(not(windows))]
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn multipart_source_cache_publication_holds_source_lock_against_concurrent_retirement() {
use crate::disk::{DiskAPI as _, disk_store::LocalDiskWrapper, os::prepared_publication_test_hooks as hooks};
let (_root, disk, _, _) = fixture().await;
std::fs::create_dir_all(disk.io_get_bucket_path(RUSTFS_META_TMP_BUCKET).expect("TMP path")).expect("TMP bucket");
let disk = Arc::new(disk);
let wrapper = LocalDiskWrapper::new(Arc::clone(&disk), false);
let upload = format!("{}/{}", "d".repeat(64), Uuid::new_v4());
let data_dir = Uuid::new_v4();
let mut init_fi = FileInfo::new(&upload, 2, 2);
init_fi.erasure.index = 1;
init_fi.data_dir = Some(data_dir);
init_fi.fresh = true;
init_fi.mod_time = Some(time::OffsetDateTime::now_utc());
let source_dir = disk
.io_get_object_path(RUSTFS_META_MULTIPART_BUCKET, &upload)
.expect("source dir");
let base_dir = disk
.io_get_bucket_path(RUSTFS_META_MULTIPART_BUCKET)
.expect("multipart bucket path");
let (entered_tx, entered_rx) = tokio::sync::oneshot::channel();
let (release_tx, release_rx) = std::sync::mpsc::channel();
let _hook = hooks::install_at(hooks::Stage::DirFsync, &source_dir, move || {
let _ = entered_tx.send(());
let _ = release_rx.recv();
});
let init_wrapper = wrapper.clone();
let init_upload = upload.clone();
let init_task = tokio::spawn(async move {
init_wrapper
.write_metadata("", RUSTFS_META_MULTIPART_BUCKET, &init_upload, init_fi)
.await
});
tokio::time::timeout(std::time::Duration::from_secs(5), entered_rx)
.await
.expect("initial write_metadata reaches parent dir fsync")
.expect("entered channel");
let retire_wrapper = wrapper.clone();
let retire_path = format!("{upload}/xl.meta");
let mut retire_fut = Box::pin(async move {
retire_wrapper
.delete_paths(RUSTFS_META_MULTIPART_BUCKET, &[retire_path])
.await
});
assert!(
futures::poll!(tokio::task::unconstrained(retire_fut.as_mut())).is_pending(),
"concurrent retirement must block on SourceRW while initial head publication is in flight"
);
let _ = release_tx.send(());
init_task.await.expect("init join").expect("initial write_metadata succeeds");
retire_fut
.await
.expect("concurrent retirement completes after SourceRW release");
assert!(
!super::super::multipart_source_state::has_active_multipart_source_head(&source_dir),
"retired source must not remain active in cache after interleaved initial publication"
);
assert!(
matches!(
super::super::multipart_source_state::validate_source_mutation(
&source_dir,
&base_dir,
&disk.publication_root,
false
),
Err(DiskError::FileNotFound)
),
"validate_source_mutation must reject non-initial mutation on retired source"
);
}
}
+76 -9
View File
@@ -401,13 +401,6 @@ impl Disk {
}
}
pub(crate) fn has_clean_active_multipart_source(&self, upload_or_part_path: &str, part_path: bool) -> bool {
match self {
Self::Local(disk) => disk.has_clean_active_multipart_source(upload_or_part_path, part_path),
Self::Remote(_) => false,
}
}
pub(crate) async fn try_publish_prepared_delete_local(
&self,
volume: &str,
@@ -667,6 +660,13 @@ impl DiskAPI for Disk {
}
}
async fn bucket_creation_witness(&self, volume: &str) -> Result<Option<os::NativeBucketCreationWitness>> {
match self {
Disk::Local(local_disk) => local_disk.bucket_creation_witness(volume).await,
Disk::Remote(_) => Ok(None),
}
}
#[tracing::instrument(level = "trace", skip_all)]
async fn make_volume(&self, volume: &str) -> Result<()> {
match self {
@@ -1524,6 +1524,11 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
// NSScanner
// Volume operations.
/// Unknown or remote backends cannot attest process-local publication history.
async fn bucket_creation_witness(&self, _volume: &str) -> Result<Option<os::NativeBucketCreationWitness>> {
Ok(None)
}
async fn make_volume(&self, volume: &str) -> Result<()>;
async fn make_volumes(&self, volume: Vec<&str>) -> Result<()>;
async fn list_volumes(&self) -> Result<Vec<VolumeInfo>>;
@@ -2299,7 +2304,7 @@ impl DeleteRecoveryJournal {
}
let id = request.version_id.unwrap_or_default();
let Some(original) = originals.get(&id) else {
return false;
return !id.is_nil() && !locals.contains_key(&id);
};
match locals.get(&id) {
Some(local) => local == original,
@@ -2349,7 +2354,10 @@ impl DeleteRecoveryJournal {
};
match (local, remote) {
(Some(local), Some(remote)) => local == remote,
(None, None) => self.original_digest.is_none() && authority.original_digest.is_none(),
(None, None) => {
(!(request.deleted && request.mark_deleted) && request.version_id.is_some_and(|id| !id.is_nil()))
|| (self.original_digest.is_none() && authority.original_digest.is_none())
}
_ => false,
}
})
@@ -3674,6 +3682,46 @@ mod delete_recovery_adoption_tests {
};
assert!(journal.matches_adoption_source(&[]), "certified non-null removal can already be absent");
assert!(journal.matches_adoption_source(std::slice::from_ref(&source)));
let missing = Uuid::new_v4();
journal.requests.push(FileInfo {
version_id: Some(missing),
..Default::default()
});
journal.original_digest = Some([3; 32]);
assert!(
journal.matches_prepared_authority(&journal),
"an explicit UUID absent from both prepared sources remains a no-op with original metadata"
);
assert!(journal.matches_adoption_source(&[]), "both non-null versions can remain absent");
assert!(
journal.matches_adoption_source(std::slice::from_ref(&source)),
"a missing non-null version in the batch remains a no-op"
);
let mut unexpected = source.clone();
unexpected.version_id = Some(missing);
let mut foreign_preparation = journal.clone();
foreign_preparation.original_versions.push(unexpected.clone());
assert!(
!journal.matches_prepared_authority(&foreign_preparation)
&& !foreign_preparation.matches_prepared_authority(&journal),
"absence from only one prepared source cannot establish a generation"
);
assert!(
!journal.matches_adoption_source(&[source.clone(), unexpected]),
"a version absent from the authority cannot be deleted locally"
);
for version_id in [None, Some(Uuid::nil())] {
journal.requests.last_mut().expect("missing-version request").version_id = version_id;
assert!(
!journal.matches_prepared_authority(&journal),
"a missing null version with original metadata cannot establish its generation"
);
assert!(
!journal.matches_adoption_source(std::slice::from_ref(&source)),
"a missing null version cannot establish its generation"
);
}
journal.requests.pop();
let mut foreign = source.clone();
foreign.metadata_digest = [2; 32];
assert!(!journal.matches_adoption_source(std::slice::from_ref(&foreign)));
@@ -3689,6 +3737,25 @@ mod delete_recovery_adoption_tests {
assert!(journal.matches_adoption_source(std::slice::from_ref(&journal.original_versions[0])));
journal.requests.clear();
assert!(!journal.matches_adoption_source(&journal.original_versions));
let mut marker = journal.clone();
marker.requests = vec![FileInfo {
version_id: Some(id),
deleted: true,
mark_deleted: true,
..Default::default()
}];
marker.original_versions.clear();
marker.original_digest = Some([3; 32]);
assert!(
!marker.matches_prepared_authority(&marker),
"a marker with original metadata requires a source generation"
);
marker.original_digest = None;
assert!(
marker.matches_prepared_authority(&marker),
"marker preparation without original metadata retains the existing allowance"
);
}
}
File diff suppressed because it is too large Load Diff
+3 -9
View File
@@ -17,8 +17,8 @@
use crate::bucket::error::BucketMetadataError;
use crate::disk::error::DiskError;
use crate::storage_api_contracts::{error::StorageErrorCode, range::HTTPRangeError};
use rustfs_s3_types::S3ErrorCode;
use rustfs_utils::path::decode_dir_object;
use s3s::S3ErrorCode;
pub type Error = StorageError;
pub type Result<T> = core::result::Result<T, Error>;
@@ -1086,14 +1086,8 @@ impl From<rmp_serde::decode::Error> for Error {
}
}
impl From<s3s::xml::SerError> for Error {
fn from(e: s3s::xml::SerError) -> Self {
Error::other(e)
}
}
impl From<s3s::xml::DeError> for Error {
fn from(e: s3s::xml::DeError) -> Self {
impl From<crate::bucket::utils::BucketConfigXmlError> for Error {
fn from(e: crate::bucket::utils::BucketConfigXmlError) -> Self {
Error::other(e)
}
}
+1 -15
View File
@@ -828,27 +828,13 @@ pub async fn create_bitrot_writer(
length: i64,
shard_size: usize,
checksum_algo: HashAlgorithm,
) -> disk::error::Result<BitrotWriterWrapper> {
create_bitrot_writer_with_orig_volume(is_inline_buffer, disk, "", volume, path, length, shard_size, checksum_algo).await
}
#[allow(clippy::too_many_arguments)]
pub async fn create_bitrot_writer_with_orig_volume(
is_inline_buffer: bool,
disk: Option<&DiskStore>,
origvolume: &str,
volume: &str,
path: &str,
length: i64,
shard_size: usize,
checksum_algo: HashAlgorithm,
) -> disk::error::Result<BitrotWriterWrapper> {
let writer = if is_inline_buffer {
CustomWriter::new_inline_buffer()
} else if let Some(disk) = disk {
let length = bitrot_create_file_size(length, shard_size, &checksum_algo);
let file = disk.create_file(origvolume, volume, path, length).await?;
let file = disk.create_file("", volume, path, length).await?;
#[cfg(feature = "hotpath")]
let file = instrument_raw_shard_writer(file, disk.is_local());
CustomWriter::new_tokio_writer(file)
@@ -333,34 +333,6 @@ struct ProofCache {
preferred: usize,
}
type VerifiedStripeKey = (uuid::Uuid, [u8; 32], u32);
static VERIFIED_STRIPE_RECORDS: std::sync::LazyLock<parking_lot::Mutex<std::collections::HashMap<VerifiedStripeKey, Bytes>>> =
std::sync::LazyLock::new(|| parking_lot::Mutex::new(std::collections::HashMap::new()));
fn get_verified_stripe_record(part: &PartIntegrity, stripe: u32) -> Option<Bytes> {
VERIFIED_STRIPE_RECORDS
.lock()
.get(&(part.generation, part.root, stripe))
.cloned()
}
fn put_verified_stripe_record(part: &PartIntegrity, stripe: u32, record: Bytes) {
let mut map = VERIFIED_STRIPE_RECORDS.lock();
if map.len() >= 4096
&& let Some(evict_key) = map.keys().next().copied()
{
map.remove(&evict_key);
}
map.insert((part.generation, part.root, stripe), record);
}
pub(crate) fn invalidate_verified_stripe_records(generation: uuid::Uuid) {
VERIFIED_STRIPE_RECORDS
.lock()
.retain(|(stored_gen, _, _), _| *stored_gen != generation);
}
impl PartProofReader {
pub(crate) async fn reconstruction_proof(
&self,
@@ -438,9 +410,6 @@ impl PartProofReader {
pub(crate) async fn record(&self, stripe: u32) -> io::Result<Bytes> {
let offset = self.part.record_offset(stripe).map_err(|_| corrupt())?;
if let Some(bytes) = get_verified_stripe_record(&self.part, stripe) {
return Ok(bytes);
}
let mut cached = self.cached.lock().await;
if let Some((_, bytes)) = cached.records.iter().find(|(number, _)| *number == stripe) {
return Ok(bytes.clone());
@@ -496,9 +465,6 @@ impl PartProofReader {
bytes
};
self.part.verify_record(stripe, &record).map_err(|_| corrupt())?;
if self.part.stripes <= 4 {
put_verified_stripe_record(&self.part, stripe, record.clone());
}
if cached.records.len() == 2 {
cached.records.pop_front();
}
@@ -897,41 +863,4 @@ mod tests {
prepared.part.verify_shard(stripe, 15, b"x", &record).expect("spilled proof");
}
}
#[tokio::test]
async fn verified_stripe_record_cache_serves_verified_record_and_still_verifies_payload() {
let layout = IntegrityLayout::new(2, 2, 8, false).expect("layout");
let mut builder = IntegrityBuilder::new(layout, 1).expect("builder");
let shards = [b"ABCD".as_slice(), b"EFGH", b"IJKL", b"MNOP"];
builder.push(shards.into_iter()).await.expect("stripe");
let prepared = builder.finish(8).await.expect("index");
let inline_bytes = prepared.inline_bytes().expect("inline bytes");
let part = prepared.part;
let warm_reader = PartProofReader {
part: part.clone(),
sources: Vec::new(),
volume: "bucket".to_owned(),
inline: Some(inline_bytes),
cached: Mutex::new(ProofCache::default()),
};
warm_reader.verify(0, 0, shards[0]).await.expect("warm verify");
assert!(
get_verified_stripe_record(&part, 0).is_some(),
"small verified stripe record should populate the verified stripe cache"
);
let cached_reader = Arc::new(PartProofReader {
part,
sources: Vec::new(),
volume: "bucket".to_owned(),
inline: None,
cached: Mutex::new(ProofCache::default()),
});
cached_reader.verify(0, 0, shards[0]).await.expect("cached verify");
assert!(
cached_reader.verify(0, 0, shards[1]).await.is_err(),
"cached record must still reject corrupt or mismatched shard payloads"
);
}
}
@@ -20,7 +20,6 @@
#![allow(clippy::all)]
use http::HeaderMap;
use s3s::dto::ETag;
use std::{collections::HashMap, io::Cursor, sync::Arc};
use tokio::io::BufReader;
@@ -29,7 +28,7 @@ use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions};
use crate::storage_api_contracts::range::HTTPRangeSpec;
use rustfs_filemeta::ObjectPartInfo;
use rustfs_rio::HashReader;
use s3s::S3ErrorCode;
use rustfs_s3_types::S3ErrorCode;
//#[derive(Clone)]
pub struct PutObjReader {
@@ -150,24 +149,6 @@ pub fn new_getobjectreader<'a>(
})
}
/// Convert a raw stored ETag into the strongly-typed `s3s::dto::ETag`.
///
/// Supports already quoted (`"abc"`), weak (`W/"abc"`), or plain (`abc`) values.
pub fn to_s3s_etag(etag: &str) -> ETag {
if let Some(rest) = etag.strip_prefix("W/\"") {
if let Some(body) = rest.strip_suffix('"') {
return ETag::Weak(body.to_string());
}
return ETag::Weak(rest.to_string());
}
if let Some(body) = etag.strip_prefix('"').and_then(|rest| rest.strip_suffix('"')) {
return ETag::Strong(body.to_string());
}
ETag::Strong(etag.to_string())
}
pub fn get_raw_etag(metadata: &HashMap<String, String>) -> String {
metadata
.get("etag")
@@ -232,28 +213,42 @@ mod tests {
}
#[test]
fn test_to_s3s_etag() {
// Test unquoted ETag - should become strong etag
assert_eq!(
to_s3s_etag("6af8d12c0c74b78094884349f3c8a079"),
ETag::Strong("6af8d12c0c74b78094884349f3c8a079".to_string())
);
fn test_new_getobjectreader_rejects_a_range_past_the_object_as_invalid_range() {
let oi = multipart_object_info();
let past_the_end = Some(HTTPRangeSpec {
is_suffix_length: false,
start: oi.size,
end: oi.size + 10,
});
assert_eq!(
to_s3s_etag("\"6af8d12c0c74b78094884349f3c8a079\""),
ETag::Strong("6af8d12c0c74b78094884349f3c8a079".to_string())
);
let err = new_getobjectreader(&past_the_end, &oi, &ObjectOptions::default(), &HeaderMap::new())
.err()
.expect("a range starting at the object size must be rejected");
assert_eq!(
to_s3s_etag("W/\"6af8d12c0c74b78094884349f3c8a079\""),
ETag::Weak("6af8d12c0c74b78094884349f3c8a079".to_string())
);
assert_eq!(err.code, S3ErrorCode::InvalidRange);
assert_eq!(err.code.as_str(), "InvalidRange");
assert_eq!(err.key, None);
assert_eq!(err.bucket_name, None);
}
assert_eq!(to_s3s_etag(""), ETag::Strong(String::new()));
#[test]
fn test_new_getobjectreader_rejects_an_unknown_size_without_a_range_as_invalid_range() {
let oi = ObjectInfo {
bucket: "bucket".to_string(),
name: "object".to_string(),
size: -1,
..Default::default()
};
assert_eq!(to_s3s_etag("\"incomplete"), ETag::Strong("\"incomplete".to_string()));
let err = new_getobjectreader(&None, &oi, &ObjectOptions::default(), &HeaderMap::new())
.err()
.expect("an object of unknown size cannot be read without a range");
assert_eq!(to_s3s_etag("incomplete\""), ETag::Strong("incomplete\"".to_string()));
assert_eq!(err.code, S3ErrorCode::InvalidRange);
assert_eq!(err.code.as_str(), "InvalidRange");
assert_eq!(err.message, "Invalid range");
assert_eq!(err.key.as_deref(), Some("object"));
assert_eq!(err.bucket_name.as_deref(), Some("bucket"));
}
#[test]
+4 -4
View File
@@ -1037,6 +1037,10 @@ pub struct ObjectOptions {
/// `ecstore` must leave it unset.
#[doc(hidden)]
pub data_movement_transferred_tier_versions: Option<crate::data_movement::TransferredTierVersions>,
/// Marks the physical copy-back write of a RestoreObject operation. This
/// is in-memory request context and must not be copied into object metadata.
#[doc(hidden)]
pub restore_copyback: bool,
/// Internal, in-memory protection context. Never populated from S3 metadata.
/// None selects the rollout default only for a new write, not for a rewrite.
#[doc(hidden)]
@@ -1543,8 +1547,6 @@ pub struct ObjectInfo {
/// published by the same upload. Callers must not repeat quota, replication,
/// lifecycle, or object-created side effects for that response.
pub multipart_completion_replayed: bool,
/// Previous current version's size observed during `rename_data` commit.
pub backfilled_old_current_size: Option<crate::disk::OldCurrentSize>,
}
impl Default for ObjectInfo {
@@ -1587,7 +1589,6 @@ impl Default for ObjectInfo {
replication_decision: String::new(),
checksum: None,
multipart_completion_replayed: false,
backfilled_old_current_size: None,
}
}
}
@@ -1631,7 +1632,6 @@ impl Clone for ObjectInfo {
replication_decision: self.replication_decision.clone(),
checksum: self.checksum.clone(),
multipart_completion_replayed: self.multipart_completion_replayed,
backfilled_old_current_size: self.backfilled_old_current_size,
expires: self.expires,
}
}
+3 -3
View File
@@ -275,16 +275,16 @@ pub(crate) type TypeLocalDiskSetDrives = Vec<Vec<Vec<Option<DiskStore>>>>;
///
/// # Returns
/// * None
pub fn set_global_region(region: s3s::region::Region) {
pub fn set_global_region(region: rustfs_s3_types::Region) {
current_ctx().set_region(region);
}
/// Get the global region
///
/// # Returns
/// * `Option<s3s::region::Region>` - The global region, if set
/// * `Option<rustfs_s3_types::Region>` - The global region, if set
///
pub fn get_global_region() -> Option<s3s::region::Region> {
pub fn get_global_region() -> Option<rustfs_s3_types::Region> {
current_ctx().region()
}
+1 -27
View File
@@ -52,7 +52,7 @@ use crate::services::event_notification::EventNotifier;
use crate::services::tier::tier::TierConfigMgr;
use crate::store::delete_recovery_runtime::DeleteRecoveryHints;
use rustfs_lock::{GlobalLockManager, get_global_lock_manager};
use s3s::region::Region;
use rustfs_s3_types::Region;
use std::collections::{HashMap, HashSet};
use std::sync::{
Arc, OnceLock,
@@ -941,32 +941,6 @@ impl InstanceContext {
.clone()
}
/// Remove the `BucketMetadataSys` when its backing `ECStore` is gone.
///
/// Called from `ECStore::drop`. When test fixtures share the bootstrap
/// context, the `BucketMetadataSys` stores a `Weak<ECStore>` to prevent a
/// reference cycle; once the store drops the weak is dangling and any
/// `object_store()` call panics. Clearing the slot here lets the next
/// fixture install a fresh `BucketMetadataSys` without hitting the stale one.
pub fn clear_bucket_metadata_sys_if_stale(&self) {
let mut guard = self.bucket_metadata_sys.lock().expect("bucket_metadata_sys lock poisoned");
if let Some(sys) = guard.as_ref() {
// `read()` may block briefly; the async RwLock is acceptable here
// because Drop is sync and the lock is held only long enough to
// check the weak reference.
let is_stale = {
let s = sys.try_read();
match s {
Ok(s) => s.is_store_stale(),
Err(_) => true, // contended: treat as stale-safe to clear
}
};
if is_stale {
*guard = None;
}
}
}
/// Set this instance's background-services cancellation token (once).
pub fn init_background_cancel_token(&self, token: CancellationToken) -> Result<(), CancellationToken> {
self.background_cancel_token.set(token)
+3 -3
View File
@@ -44,10 +44,10 @@ use crate::{
};
use rustfs_concurrency::WorkloadAdmissionSnapshotProvider;
use rustfs_config::server_config::{Config, get_global_server_config, set_global_server_config};
use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
use rustfs_io_metrics::internode_metrics::global_internode_metrics;
use rustfs_lock::client::LockClient;
use s3s::dto::BucketLifecycleConfiguration;
use s3s::region::Region;
use rustfs_s3_types::Region;
use tokio::sync::{OwnedRwLockReadGuard, RwLock};
use tokio_util::sync::CancellationToken;
#[cfg(test)]
@@ -371,7 +371,7 @@ pub fn global_tier_config_mgr() -> Arc<RwLock<TierConfigMgr>> {
get_global_tier_config_mgr()
}
pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<BucketLifecycleConfiguration> {
pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<PersistedLifecycleConfiguration> {
GLOBAL_LIFECYCLE_SYS.get(bucket).await
}
@@ -703,6 +703,28 @@ pub async fn acquire_legacy_transition_state_reconcile_fleet_proof() -> Option<L
.then_some(proof)
}
pub async fn require_restore_copyback_fleet_capability() -> Result<()> {
let Some(topology) = REMOTE_VERSION_STATE_PROBE_TOPOLOGY.get() else {
#[cfg(any(test, feature = "test-util"))]
return Ok(());
#[cfg(not(any(test, feature = "test-util")))]
return Err(Error::other("restore copy-back capability topology is unavailable"));
};
let Some(notification_sys) = get_global_notification_sys() else {
#[cfg(any(test, feature = "test-util"))]
return Ok(());
#[cfg(not(any(test, feature = "test-util")))]
return Err(Error::other("restore copy-back capability probe is unavailable"));
};
timeout(
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
notification_sys.probe_restore_copyback_fleet(topology),
)
.await
.map_err(|_| Error::other("restore copy-back capability fleet probe timed out"))??;
Ok(())
}
fn acquire_legacy_transition_state_reconcile_fleet_proof_from(
state: &FleetCapabilityProofState,
expected_topology: &str,
@@ -1760,6 +1782,30 @@ impl NotificationSys {
Ok(peer_epochs)
}
async fn probe_restore_copyback_fleet(&self, topology_fingerprint: &str) -> Result<BTreeMap<String, Uuid>> {
if self.peer_clients.len() != self.peer_topology_hosts.len() {
return Err(Error::other("restore copy-back capability fleet membership is incomplete"));
}
let probes = self.peer_clients.iter().map(|client| async {
let client = client
.as_ref()
.ok_or_else(|| Error::other("restore copy-back capability peer is unreachable"))?;
client
.probe_restore_copyback_capability(topology_fingerprint.to_string())
.await
});
let mut peer_epochs = BTreeMap::new();
for result in join_all(probes).await {
let (peer, epoch) = result?;
insert_remote_version_state_peer(&mut peer_epochs, peer, epoch)?;
}
let expected = self.peer_topology_hosts.iter().cloned().collect::<BTreeSet<_>>();
if expected.len() != self.peer_topology_hosts.len() || peer_epochs.keys().ne(expected.iter()) {
return Err(Error::other("restore copy-back capability fleet membership does not match topology"));
}
Ok(peer_epochs)
}
async fn probe_transition_transaction_compaction_fleet(&self, topology_fingerprint: &str) -> Result<BTreeMap<String, Uuid>> {
if self.peer_clients.len() != self.peer_topology_hosts.len() {
return Err(Error::other(
+19 -18
View File
@@ -1032,8 +1032,9 @@ mod tests {
use crate::storage_api_contracts::object::ObjectIO as _;
use http::HeaderMap;
use rustfs_filemeta::{FileInfo, FileMeta, ObjectPartInfo, TransitionVersionState};
use s3s::dto::{
BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, NoncurrentVersionExpiration,
use rustfs_gateway_types::persistence::{
PersistedLifecycleConfiguration, PersistedLifecycleExpiration, PersistedLifecycleRule,
PersistedNoncurrentVersionExpiration,
};
use std::time::Duration as StdDuration;
use time::OffsetDateTime;
@@ -1077,12 +1078,12 @@ mod tests {
}
}
fn expired_delete_marker_lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn expired_delete_marker_lifecycle() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
expired_object_delete_marker: Some(true),
..Default::default()
}),
@@ -2110,11 +2111,11 @@ mod tests {
))
.await
.expect("seed the expired source version");
let lifecycle = BucketLifecycleConfiguration {
let lifecycle = PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
expiration: Some(LifecycleExpiration {
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: Some(PersistedLifecycleExpiration {
days: Some(1),
expired_object_all_versions: Some(true),
..Default::default()
@@ -2363,7 +2364,7 @@ mod tests {
versioned,
version_id,
transition: crate::bucket::lifecycle::lifecycle::TransitionOptions {
restore_request: s3s::dto::RestoreRequest {
restore_request: rustfs_lifecycle::RestoreRequest {
days: Some(1),
..Default::default()
},
@@ -2443,7 +2444,7 @@ mod tests {
async fn run_entry_with(
&self,
entry: MetaCacheEntry,
lifecycle_config: Option<BucketLifecycleConfiguration>,
lifecycle_config: Option<PersistedLifecycleConfiguration>,
) -> RebalanceEntryOutcome {
Box::pin(Arc::clone(&self.store).rebalance_entry(
RebalanceEntryTarget {
@@ -2779,17 +2780,17 @@ mod tests {
}
/// A rule that expires every noncurrent version at once.
fn noncurrent_expire_all_lifecycle() -> BucketLifecycleConfiguration {
BucketLifecycleConfiguration {
fn noncurrent_expire_all_lifecycle() -> PersistedLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("noncurrent-expire-all".to_string()),
noncurrent_version_expiration: Some(NoncurrentVersionExpiration {
noncurrent_version_expiration: Some(PersistedNoncurrentVersionExpiration {
noncurrent_days: Some(0),
newer_noncurrent_versions: None,
}),
@@ -67,8 +67,8 @@ use crate::error::{Error, Result};
use crate::storage_api_contracts::range::HTTPRangeSpec;
use rustfs_filemeta::TRANSITION_COMPLETE;
use rustfs_filemeta::{FileInfo, MetaCacheEntry};
use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
use rustfs_rio::Index;
use s3s::dto::ReplicationConfiguration;
use serde::Serialize;
use std::io::Cursor;
use std::sync::Arc;
@@ -2676,7 +2676,7 @@ fn test_resolve_rebalance_optional_bucket_config_result_passthrough() {
let result = resolve_rebalance_optional_bucket_config_result(
"bucket-a",
"replication",
Ok((ReplicationConfiguration::default(), OffsetDateTime::UNIX_EPOCH)),
Ok((PersistedReplicationConfiguration::default(), OffsetDateTime::UNIX_EPOCH)),
)
.expect("bucket config should pass through");
assert!(result.is_some());
@@ -34,9 +34,9 @@ pub struct RebalanceStats {
#[derive(Debug, Default)]
pub(super) struct RebalanceBucketConfigs {
pub(super) bucket_incarnation_id: Option<uuid::Uuid>,
pub(super) lifecycle_config: Option<s3s::dto::BucketLifecycleConfiguration>,
pub(super) object_lock_config: Option<s3s::dto::ObjectLockConfiguration>,
pub(super) replication_config: Option<(s3s::dto::ReplicationConfiguration, OffsetDateTime)>,
pub(super) lifecycle_config: Option<rustfs_gateway_types::persistence::PersistedLifecycleConfiguration>,
pub(super) object_lock_config: Option<rustfs_gateway_types::persistence::PersistedObjectLockConfiguration>,
pub(super) replication_config: Option<(rustfs_gateway_types::persistence::PersistedReplicationConfiguration, OffsetDateTime)>,
}
#[derive(Debug, Clone, PartialEq, Eq)]
+94 -43
View File
@@ -80,10 +80,11 @@ use crate::{
store::ECStore,
};
use rustfs_filemeta::FileInfo;
use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
use rustfs_rio::HashReader;
use rustfs_s3_client::{admin_handler_utils::AdminError, provider_versions::ProviderVersionCapabilities};
use rustfs_s3_types::S3ErrorCode;
use rustfs_utils::path::{SLASH_SEPARATOR, path_join};
use s3s::{S3ErrorCode, dto::BucketLifecycleConfiguration};
use super::{
tier_handlers::{ERR_TIER_BUCKET_NOT_FOUND, ERR_TIER_CONNECT_ERR, ERR_TIER_INVALID_CREDENTIALS, ERR_TIER_PERM_ERR},
@@ -1102,7 +1103,7 @@ trait TierReferenceProofStore:
.await
}
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>>;
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<PersistedLifecycleConfiguration>>;
async fn walk_bucket_for_deletion_check(
self: Arc<Self>,
@@ -1155,7 +1156,7 @@ impl TierReferenceProofStore for ECStore {
Ok(buckets.into_values().collect())
}
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<PersistedLifecycleConfiguration>> {
match get_lifecycle_config(bucket).await {
Ok((config, _updated_at)) => Ok(Some(config)),
Err(Error::ConfigNotFound) => Ok(None),
@@ -1634,7 +1635,7 @@ struct TierLifecycleReference<'a> {
}
fn lifecycle_config_target_reference<'a>(
config: &'a BucketLifecycleConfiguration,
config: &'a PersistedLifecycleConfiguration,
targets: &[TierMutationIntentTarget],
) -> Option<TierLifecycleReference<'a>> {
for rule in &config.rules {
@@ -7261,9 +7262,8 @@ mod tests {
endpoints::{Endpoints, PoolEndpoints, SetupType},
};
use crate::services::tier::tier_mutation_intent::TIER_MUTATION_INTENT_RECORD_PREFIX;
use s3s::dto::{
BucketLifecycleConfiguration, ExpirationStatus, LifecycleRule, NoncurrentVersionTransition, Transition,
TransitionStorageClass,
use rustfs_gateway_types::persistence::{
PersistedLifecycleConfiguration, PersistedLifecycleRule, PersistedNoncurrentVersionTransition, PersistedTransition,
};
#[test]
@@ -8090,7 +8090,7 @@ mod tests {
#[async_trait::async_trait]
impl TierReferenceProofStore for LockingTierConfigStore {
async fn lifecycle_config_for_reference_proof(&self, _bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
async fn lifecycle_config_for_reference_proof(&self, _bucket: &str) -> Result<Option<PersistedLifecycleConfiguration>> {
Ok(None)
}
}
@@ -10730,6 +10730,42 @@ mod tests {
assert_eq!(observed.committed_config_etag.as_deref(), Some("etag-new"));
}
#[tokio::test]
async fn intent_advance_retries_cas_conflict_reload_lock_timeout() {
let store = Arc::new(CasConfigStore::default());
let mutation_id = uuid::Uuid::from_u128(39);
let prepared = prepared_remove_intent("COLD-A", mutation_id);
crate::services::tier::tier_mutation_intent::save_tier_mutation_intent_record(store.clone(), &prepared)
.await
.expect("prepared intent fixture should persist");
let committed = committed_remove_intent("COLD-A", mutation_id, "etag-new");
let committed_bytes = committed.encode().expect("committed intent fixture should encode");
let object = crate::services::tier::tier_mutation_intent::tier_mutation_intent_record_object_name(mutation_id)
.expect("record object should build");
let winning_etag = format!("etag-{}", store.next_etag.load(Ordering::SeqCst) + 1);
store.rewrite_on_next_if_match(object.clone(), committed_bytes.clone()).await;
*store.read_timeout_after_if_match_race.lock().await = Some(object.clone());
let (observed, applied) = crate::services::tier::tier_mutation_intent::advance_tier_mutation_intent_record_idempotent(
store.clone(),
mutation_id,
TierMutationIntentState::Committed,
Some("etag-new".to_string()),
)
.await
.expect("CAS conflict reload should retry its typed lock timeout");
assert!(!applied);
assert_eq!(observed, committed);
assert!(store.read_timeout_after_if_match_race.lock().await.is_none());
assert_eq!(
store.objects.lock().await.get(&object).cloned(),
Some((committed_bytes, winning_etag)),
"retry must preserve the complete winning record and its object ETag"
);
}
#[tokio::test]
async fn intent_advance_reconciles_matching_commit_after_final_cas_race() {
let store = Arc::new(CasConfigStore::default());
@@ -15230,6 +15266,7 @@ mod tests {
objects: tokio::sync::Mutex<HashMap<String, (Vec<u8>, String)>>,
legacy_state: tokio::sync::Mutex<Option<Vec<u8>>>,
if_match_race_rewrite: tokio::sync::Mutex<std::collections::VecDeque<(String, Vec<u8>)>>,
read_timeout_after_if_match_race: tokio::sync::Mutex<Option<String>>,
next_etag: AtomicUsize,
content_derived_etags: AtomicBool,
fail_put: AtomicBool,
@@ -15248,7 +15285,7 @@ mod tests {
lock_manager: Arc<rustfs_lock::GlobalLockManager>,
lock_requests: Mutex<Vec<(String, String)>>,
listed_versions: Mutex<Vec<ObjectInfo>>,
lifecycle_configs: Mutex<HashMap<String, BucketLifecycleConfiguration>>,
lifecycle_configs: Mutex<HashMap<String, PersistedLifecycleConfiguration>>,
}
impl Default for CasConfigStore {
@@ -15259,6 +15296,7 @@ mod tests {
objects: tokio::sync::Mutex::new(HashMap::new()),
legacy_state: tokio::sync::Mutex::new(None),
if_match_race_rewrite: tokio::sync::Mutex::new(std::collections::VecDeque::new()),
read_timeout_after_if_match_race: tokio::sync::Mutex::new(None),
next_etag: AtomicUsize::new(0),
content_derived_etags: AtomicBool::new(false),
fail_put: AtomicBool::new(false),
@@ -15311,7 +15349,7 @@ mod tests {
.retain(|version| version.bucket != bucket || version.name != object);
}
fn add_lifecycle_config(&self, bucket: &str, config: BucketLifecycleConfiguration) {
fn add_lifecycle_config(&self, bucket: &str, config: PersistedLifecycleConfiguration) {
self.lifecycle_configs
.lock()
.expect("tier reference fixture should not poison")
@@ -15420,6 +15458,19 @@ mod tests {
crate::api::diagnostic::current_read_version_entry_source(),
rustfs_io_metrics::s3_http_metrics::current_s3_http_operation_name(),
));
{
// The queued rewrite keeps the initial read successful. Only
// the first target read after that CAS race consumes the fault.
let race_completed = self.if_match_race_rewrite.lock().await.is_empty();
let mut timeout = self.read_timeout_after_if_match_race.lock().await;
if race_completed && timeout.as_deref() == Some(object) {
timeout.take();
return Err(Error::Lock(rustfs_lock::LockError::Timeout {
resource: format!("{bucket}/{object}@latest"),
timeout: std::time::Duration::from_secs(5),
}));
}
}
let legacy_config_path = tier_config_path(TIER_CONFIG_LEGACY_FILE);
let (data, etag) = if object == legacy_config_path {
let state = self.legacy_state.lock().await;
@@ -16048,7 +16099,7 @@ mod tests {
.await
}
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<BucketLifecycleConfiguration>> {
async fn lifecycle_config_for_reference_proof(&self, bucket: &str) -> Result<Option<PersistedLifecycleConfiguration>> {
if self.reference_lifecycle_error.load(Ordering::SeqCst) {
return Err(Error::CorruptedFormat);
}
@@ -16183,10 +16234,10 @@ mod tests {
});
store.add_lifecycle_config(
"photos",
BucketLifecycleConfiguration {
PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
@@ -16195,10 +16246,10 @@ mod tests {
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: Some(vec![Transition {
transitions: Some(vec![PersistedTransition {
days: Some(1),
date: None,
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
storage_class: Some("COLD-A".to_owned()),
}]),
}],
},
@@ -16851,10 +16902,10 @@ mod tests {
.save_tiering_config_if_current(store.clone(), None)
.await
.expect("reference proof fixture should persist");
let config = BucketLifecycleConfiguration {
let config = PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
@@ -16863,10 +16914,10 @@ mod tests {
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: Some(vec![Transition {
transitions: Some(vec![PersistedTransition {
days: Some(1),
date: None,
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
storage_class: Some("COLD-A".to_owned()),
}]),
}],
};
@@ -17296,23 +17347,23 @@ mod tests {
});
store.add_lifecycle_config(
"private-bucket",
BucketLifecycleConfiguration {
rustfs_gateway_types::persistence::PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
rules: vec![rustfs_gateway_types::persistence::PersistedLifecycleRule {
status: "Enabled".to_string(),
id: Some("private-rule".to_string()),
noncurrent_version_transitions: noncurrent.then(|| {
vec![NoncurrentVersionTransition {
vec![rustfs_gateway_types::persistence::PersistedNoncurrentVersionTransition {
noncurrent_days: Some(1),
newer_noncurrent_versions: None,
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
storage_class: Some("COLD-A".to_string()),
}]
}),
transitions: (!noncurrent).then(|| {
vec![Transition {
vec![rustfs_gateway_types::persistence::PersistedTransition {
days: Some(1),
date: None,
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
storage_class: Some("COLD-A".to_string()),
}]
}),
expiration: None,
@@ -17715,10 +17766,10 @@ mod tests {
old_backend_identity: Some(current_identity),
new_backend_identity: None,
};
let config = BucketLifecycleConfiguration {
let config = PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
@@ -17727,10 +17778,10 @@ mod tests {
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: Some(vec![Transition {
transitions: Some(vec![PersistedTransition {
days: Some(1),
date: None,
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
storage_class: Some("COLD-A".to_owned()),
}]),
}],
};
@@ -17760,20 +17811,20 @@ mod tests {
old_backend_identity: Some(current_identity),
new_backend_identity: None,
};
let config = BucketLifecycleConfiguration {
let config = PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
filter: None,
id: Some("move-noncurrent".to_string()),
noncurrent_version_expiration: None,
noncurrent_version_transitions: Some(vec![NoncurrentVersionTransition {
noncurrent_version_transitions: Some(vec![PersistedNoncurrentVersionTransition {
noncurrent_days: Some(1),
newer_noncurrent_versions: None,
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
storage_class: Some("COLD-A".to_owned()),
}]),
prefix: None,
transitions: None,
@@ -17809,10 +17860,10 @@ mod tests {
old_backend_identity: Some(current_identity),
new_backend_identity: None,
};
let config = BucketLifecycleConfiguration {
let config = PersistedLifecycleConfiguration {
expiry_updated_at: None,
rules: vec![LifecycleRule {
status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
rules: vec![PersistedLifecycleRule {
status: "Enabled".to_owned(),
expiration: None,
abort_incomplete_multipart_upload: None,
del_marker_expiration: None,
@@ -17821,10 +17872,10 @@ mod tests {
noncurrent_version_expiration: None,
noncurrent_version_transitions: None,
prefix: None,
transitions: Some(vec![Transition {
transitions: Some(vec![PersistedTransition {
days: Some(1),
date: None,
storage_class: Some(TransitionStorageClass::from_static("COLD-A")),
storage_class: Some("COLD-A".to_owned()),
}]),
}],
};
@@ -700,9 +700,25 @@ async fn advance_tier_mutation_intent_record_idempotent_at_prefix<S>(
where
S: EcstoreObjectIO,
{
let mut original_intent: Option<TierMutationIntent> = None;
for attempt in 0..TIER_MUTATION_INTENT_ADVANCE_CAS_ATTEMPTS {
let (mut intent, current_etag) =
load_tier_mutation_intent_record_with_etag_at_prefix(api.clone(), prefix, mutation_id).await?;
match load_tier_mutation_intent_record_with_etag_at_prefix(api.clone(), prefix, mutation_id).await {
Ok(current) => current,
Err(Error::Lock(rustfs_lock::LockError::Timeout { .. }))
if attempt + 1 < TIER_MUTATION_INTENT_ADVANCE_CAS_ATTEMPTS =>
{
continue;
}
Err(err) => return Err(err),
};
if let Some(original) = &original_intent {
if !intent.same_identity_as(original) {
return Err(Error::PreconditionFailed);
}
} else {
original_intent = Some(intent.clone());
}
let advanced = intent
.advance_idempotent(next, committed_config_etag.clone())
.map_err(tier_mutation_intent_store_error)?;
@@ -713,7 +729,15 @@ where
Ok(()) => return Ok((intent, true)),
Err(Error::PreconditionFailed) => {
let (mut current, _) =
load_tier_mutation_intent_record_with_etag_at_prefix(api.clone(), prefix, mutation_id).await?;
match load_tier_mutation_intent_record_with_etag_at_prefix(api.clone(), prefix, mutation_id).await {
Ok(current) => current,
Err(Error::Lock(rustfs_lock::LockError::Timeout { .. }))
if attempt + 1 < TIER_MUTATION_INTENT_ADVANCE_CAS_ATTEMPTS =>
{
continue;
}
Err(err) => return Err(err),
};
if !current.same_identity_as(&intent) {
return Err(Error::PreconditionFailed);
}
@@ -728,6 +752,11 @@ where
}
return Ok((current, false));
}
Err(Error::Lock(rustfs_lock::LockError::Timeout { .. }))
if attempt + 1 < TIER_MUTATION_INTENT_ADVANCE_CAS_ATTEMPTS =>
{
continue;
}
Err(err) => return Err(err),
}
}
@@ -36,7 +36,9 @@ use crate::services::tier::{
use bytes::Bytes;
use http::StatusCode;
use rustfs_filemeta::metadata_keys;
use rustfs_gateway_types::dto::{ObjectLockLegalHoldStatus, ObjectLockMode, ReplicationStatus};
use rustfs_s3_client::credentials::{Credentials, SignatureType, Static, Value};
use rustfs_s3_client::remote_error_code::RemoteErrorCode;
use rustfs_s3_client::transition_api::{BucketLookupType, Options, TransitionClient, TransitionClientTimeouts, TransitionCore};
use rustfs_s3_client::{
admin_handler_utils::AdminError,
@@ -49,10 +51,6 @@ use rustfs_utils::egress::validate_outbound_url;
use rustfs_utils::http::headers::{
CACHE_CONTROL, CONTENT_DISPOSITION, CONTENT_ENCODING, CONTENT_LANGUAGE, CONTENT_TYPE, EXPIRES, HeaderExt as _,
};
use s3s::{
S3ErrorCode,
dto::{ObjectLockLegalHoldStatus, ObjectLockRetentionMode, ReplicationStatus},
};
use std::collections::HashMap;
use std::sync::Arc;
use std::time::Duration;
@@ -193,12 +191,15 @@ pub trait WarmBackend {
.await
{
Ok(_) => Ok(TransitionCandidateProbe::VersionedPresent(remote_version_id.to_string())),
Err(err) if matches!(to_error_response(&err).code, S3ErrorCode::InvalidRange) => {
Err(err) if matches!(to_error_response(&err).code, RemoteErrorCode::INVALID_RANGE) => {
Ok(TransitionCandidateProbe::VersionedPresent(remote_version_id.to_string()))
}
Err(err)
if err.kind() == std::io::ErrorKind::NotFound
|| matches!(to_error_response(&err).code, S3ErrorCode::NoSuchKey | S3ErrorCode::NoSuchVersion) =>
|| matches!(
to_error_response(&err).code,
RemoteErrorCode::NO_SUCH_KEY | RemoteErrorCode::NO_SUCH_VERSION
) =>
{
Ok(TransitionCandidateProbe::Missing)
}
@@ -218,9 +219,9 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap
let mut opts = PutObjectOptions {
storage_class,
send_content_md5: true,
legalhold: ObjectLockLegalHoldStatus::from_static(""),
legalhold: ObjectLockLegalHoldStatus::custom(""),
internal: AdvancedPutOptions {
replication_status: ReplicationStatus::from_static(""),
replication_status: ReplicationStatus::custom(""),
..Default::default()
},
..Default::default()
@@ -251,7 +252,7 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap
}
if let Some(mode) = metadata.lookup(metadata_keys::OBJECT_LOCK_MODE) {
opts.mode = ObjectLockRetentionMode::from(mode.to_ascii_uppercase());
opts.mode = ObjectLockMode::custom(mode.to_ascii_uppercase());
}
if let Some(retain_until_date) = metadata
@@ -262,7 +263,7 @@ pub fn build_transition_put_options(storage_class: String, mut metadata: HashMap
}
if let Some(legalhold) = metadata.lookup(metadata_keys::OBJECT_LOCK_LEGAL_HOLD) {
opts.legalhold = ObjectLockLegalHoldStatus::from(legalhold.to_ascii_uppercase());
opts.legalhold = ObjectLockLegalHoldStatus::custom(legalhold.to_ascii_uppercase());
}
// Promoted keys are read above through `lookup`, which accepts more than
@@ -679,7 +680,7 @@ fn is_remote_tier_auth_rejection(err: &std::io::Error) -> bool {
matches!(response.status_code, StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN)
|| matches!(
response.code,
S3ErrorCode::AccessDenied | S3ErrorCode::InvalidAccessKeyId | S3ErrorCode::SignatureDoesNotMatch
RemoteErrorCode::ACCESS_DENIED | RemoteErrorCode::INVALID_ACCESS_KEY_ID | RemoteErrorCode::SIGNATURE_DOES_NOT_MATCH
)
}
@@ -1304,7 +1305,7 @@ mod tests {
struct AuthRejectingProbePutBackend {
status_code: StatusCode,
code: S3ErrorCode,
code: RemoteErrorCode,
probes: Arc<AtomicUsize>,
removes: Arc<AtomicUsize>,
}
@@ -1767,10 +1768,10 @@ mod tests {
#[tokio::test]
async fn check_warm_backend_reports_auth_rejected_probe_put_as_invalid_credentials() {
for (status_code, code) in [
(StatusCode::FORBIDDEN, S3ErrorCode::AccessDenied),
(StatusCode::UNAUTHORIZED, S3ErrorCode::Custom("".into())),
(StatusCode::OK, S3ErrorCode::InvalidAccessKeyId),
(StatusCode::OK, S3ErrorCode::SignatureDoesNotMatch),
(StatusCode::FORBIDDEN, RemoteErrorCode::ACCESS_DENIED),
(StatusCode::UNAUTHORIZED, RemoteErrorCode::custom("")),
(StatusCode::OK, RemoteErrorCode::INVALID_ACCESS_KEY_ID),
(StatusCode::OK, RemoteErrorCode::SIGNATURE_DOES_NOT_MATCH),
] {
let probes = Arc::new(AtomicUsize::new(0));
let removes = Arc::new(AtomicUsize::new(0));
@@ -2165,18 +2166,31 @@ mod tests {
metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(),
ObjectLockLegalHoldStatus::ON.to_string(),
);
metadata.insert(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
ObjectLockRetentionMode::GOVERNANCE.to_string(),
);
metadata.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), ObjectLockMode::GOVERNANCE.to_string());
let opts = build_transition_put_options("COLD".to_string(), metadata);
assert_eq!(opts.mode.as_str(), ObjectLockRetentionMode::GOVERNANCE);
assert_eq!(opts.legalhold.as_str(), ObjectLockLegalHoldStatus::ON);
assert_eq!(opts.mode, ObjectLockMode::GOVERNANCE);
assert_eq!(opts.legalhold, ObjectLockLegalHoldStatus::ON);
assert_ne!(opts.retain_until_date, OffsetDateTime::UNIX_EPOCH);
}
#[test]
fn build_transition_put_options_sends_no_object_lock_or_replication_headers_by_default() {
let opts = build_transition_put_options("COLD".to_string(), HashMap::new());
assert_eq!(opts.mode.as_str(), "");
assert_eq!(opts.legalhold.as_str(), "");
assert_eq!(opts.internal.replication_status.as_str(), "");
let header = opts.header();
for name in [
"x-amz-object-lock-mode",
"x-amz-object-lock-legal-hold",
"x-amz-replication-status",
] {
assert!(header.get(name).is_none(), "{name} must not be sent: {header:?}");
}
}
#[test]
fn build_transition_put_options_filters_promoted_headers_from_user_metadata() {
let mut metadata = HashMap::new();
@@ -28,6 +28,8 @@ use crate::services::tier::{
},
};
use http::HeaderMap;
use rustfs_gateway_types::dto::Status as BucketVersioningStatus;
use rustfs_s3_client::remote_error_code::RemoteErrorCode;
use rustfs_s3_client::{
api_error_response::to_error_response,
api_get_options::GetObjectOptions,
@@ -42,7 +44,6 @@ use rustfs_s3_client::{
};
use rustfs_utils::egress::validate_outbound_url_allow_private;
use rustfs_utils::path::SLASH_SEPARATOR;
use s3s::{S3ErrorCode, dto::BucketVersioningStatus};
pub struct WarmBackendS3 {
pub client: Arc<TransitionClient>,
@@ -61,8 +62,8 @@ enum RemoteBucketVersioning {
fn remote_bucket_versioning_from_status(status: Option<&str>) -> Result<RemoteBucketVersioning, std::io::Error> {
Ok(match status {
Some(BucketVersioningStatus::ENABLED) => RemoteBucketVersioning::Enabled,
Some(BucketVersioningStatus::SUSPENDED) => RemoteBucketVersioning::Suspended,
Some(status) if status == BucketVersioningStatus::ENABLED.as_str() => RemoteBucketVersioning::Enabled,
Some(status) if status == BucketVersioningStatus::SUSPENDED.as_str() => RemoteBucketVersioning::Suspended,
Some(status) => {
return Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
@@ -237,7 +238,7 @@ impl WarmBackendS3 {
}
Err(err) => {
let response = to_error_response(&err);
if response.code == S3ErrorCode::NoSuchKey {
if response.code == RemoteErrorCode::NO_SUCH_KEY {
Ok(TransitionCandidateProbe::Missing)
} else {
Err(err)
@@ -324,7 +325,7 @@ impl WarmBackendS3 {
}
Err(err) => {
let response = to_error_response(&err);
if response.code == S3ErrorCode::NoSuchKey {
if response.code == RemoteErrorCode::NO_SUCH_KEY {
Ok(TransitionCandidateProbe::Missing)
} else {
Err(err)
@@ -688,7 +689,7 @@ mod tests {
.probe_transition_candidate("forbidden-probe")
.await
.expect_err("an authorization failure must not be mistaken for a missing key");
assert_eq!(to_error_response(&err).code, S3ErrorCode::AccessDenied);
assert_eq!(to_error_response(&err).code, RemoteErrorCode::ACCESS_DENIED);
assert_eq!(
backend
.probe_transition_candidate("delete-marker-hidden")
@@ -1066,11 +1067,13 @@ mod tests {
RemoteBucketVersioning::Disabled
);
assert_eq!(
remote_bucket_versioning_from_status(Some(BucketVersioningStatus::ENABLED)).expect("enabled status should parse"),
remote_bucket_versioning_from_status(Some(BucketVersioningStatus::ENABLED.as_str()))
.expect("enabled status should parse"),
RemoteBucketVersioning::Enabled
);
assert_eq!(
remote_bucket_versioning_from_status(Some(BucketVersioningStatus::SUSPENDED)).expect("suspended status should parse"),
remote_bucket_versioning_from_status(Some(BucketVersioningStatus::SUSPENDED.as_str()))
.expect("suspended status should parse"),
RemoteBucketVersioning::Suspended
);
let err = remote_bucket_versioning_from_status(Some("UnexpectedStatus"))
@@ -18,7 +18,6 @@ use std::{
sync::atomic::{AtomicBool, Ordering},
};
use s3s::header::{X_AMZ_DELETE_MARKER, X_AMZ_VERSION_ID};
use uuid::Uuid;
use crate::services::tier::{
@@ -128,7 +127,7 @@ impl WarmBackend for WarmBackendWasabi {
self.ensure_unversioned().await?;
}
let (headers, reader) = self.s3.get_with_headers(object, rv, opts).await?;
if rv.is_empty() && (headers.contains_key(X_AMZ_VERSION_ID) || headers.contains_key(X_AMZ_DELETE_MARKER)) {
if rv.is_empty() && (headers.contains_key("x-amz-version-id") || headers.contains_key("x-amz-delete-marker")) {
self.versioning_drifted.store(true, Ordering::Release);
return Err(io::Error::other(
"Wasabi bucket versioning changed while reading an unversioned tier object",
@@ -5340,14 +5340,6 @@ impl SetDisks {
}
let disks = self.get_disks_internal().await;
if disks.len() == self.set_drive_count
&& disks.iter().all(|disk| {
disk.as_ref()
.is_some_and(|disk| disk.has_clean_active_multipart_source(dst_object, true))
})
{
return Ok(false);
}
let transaction_path = part_transaction_path(dst_object);
let transaction_meta_path = format!("{transaction_path}/{PART_TRANSACTION_NEW_META}");
@@ -5547,14 +5539,6 @@ impl SetDisks {
write_quorum: usize,
) -> disk::error::Result<()> {
let disks = self.get_disks_internal().await;
if disks.len() == self.set_drive_count
&& disks.iter().all(|disk| {
disk.as_ref()
.is_some_and(|disk| disk.has_clean_active_multipart_source(part_path, false))
})
{
return Ok(());
}
let listings = join_all(disks.iter().map(|disk| {
let disk = disk.clone();
async move {
@@ -5614,11 +5598,10 @@ impl SetDisks {
write_quorum: usize,
quorum_context: Option<MultipartWriteQuorumContext<'_>>,
) -> disk::error::Result<Vec<Option<DiskStore>>> {
self.recover_part_transaction(dst_object, write_quorum).await?;
let part = ObjectPartInfo::unmarshal(&meta)?;
let integrity = part.integrity.map(Arc::new);
self.recover_part_transaction(dst_object, write_quorum).await?;
let src_bucket = Arc::new(src_bucket.to_string());
let src_object = Arc::new(src_object.to_string());
let dst_bucket = Arc::new(dst_bucket.to_string());
@@ -5640,9 +5623,7 @@ impl SetDisks {
if let Err(error) = prepared {
return Some(Err(error));
}
if let Some(integrity) = integrity
&& !disk.is_local()
{
if let Some(integrity) = integrity {
let Some((directory, _)) = dst_object.rsplit_once('/') else {
return Some(Err(DiskError::FileCorrupt));
};
@@ -5759,36 +5740,7 @@ impl SetDisks {
return Err(err);
}
let all_prepared_succeeded = errs
.iter()
.zip(prepared_disks.iter())
.all(|(err, disk)| err.is_none() && disk.as_ref().is_some_and(|d| d.is_local()));
let committed = if all_prepared_succeeded {
let settles = prepared_disks.iter().map(|disk| {
let disk = disk.clone();
let dst_object = dst_object.clone();
async move {
let Some(disk) = disk else {
return Err(DiskError::DiskNotFound);
};
disk.settle_part_transaction(RUSTFS_META_MULTIPART_BUCKET, &dst_object, PartTransactionAction::Commit)
.await
}
});
let settle_results = join_all(settles).await;
if settle_results.iter().filter(|res| res.is_ok()).count() >= write_quorum
&& settle_results
.iter()
.zip(prepared_disks.iter())
.all(|(res, disk)| disk.is_none() || res.is_ok())
{
true
} else {
self.recover_part_transaction(&dst_object, write_quorum).await?
}
} else {
self.recover_part_transaction(&dst_object, write_quorum).await?
};
let committed = self.recover_part_transaction(&dst_object, write_quorum).await?;
if !committed {
let err = DiskError::ErasureWriteQuorum;
if let Some(context) = quorum_context {
@@ -6785,7 +6737,9 @@ impl SetDisks {
object,
);
}
if should_prevent_write(&oi, http_preconditions.if_none_match_value(), http_preconditions.if_match_value())
let if_none_match = http_preconditions.if_none_match_value().map(str::to_owned);
let if_match = http_preconditions.if_match_value().map(str::to_owned);
if should_prevent_write(&oi, if_none_match, if_match)
&& !crate::data_movement::can_replace_stale_data_movement_target(&oi, opts)
{
return Some(StorageError::PreconditionFailed);
+5 -15
View File
@@ -161,7 +161,7 @@ impl SetDisks {
)
});
let mut metadata = FileMeta::new();
for (version_id, parts) in versions {
for parts in versions.into_values() {
let errors = parts
.iter()
.zip(&missing)
@@ -173,25 +173,15 @@ impl SetDisks {
}
})
.collect::<Vec<_>>();
let (read_quorum, write_quorum) = match Self::object_quorum_from_meta(&parts, &errors, default_parity) {
Ok((read, write)) => (
usize::try_from(read).map_err(|_| DiskError::FileCorrupt)?,
usize::try_from(write).map_err(|_| DiskError::FileCorrupt)?,
),
let read_quorum = match Self::object_quorum_from_meta(&parts, &errors, default_parity) {
Ok((quorum, _)) => usize::try_from(quorum).map_err(|_| DiskError::FileCorrupt)?,
Err(DiskError::FileNotFound | DiskError::FileVersionNotFound) if complete => continue,
Err(DiskError::FileNotFound | DiskError::FileVersionNotFound) => return Err(DiskError::ErasureReadQuorum),
Err(DiskError::ErasureReadQuorum) if complete => continue,
Err(error) => return Err(error),
};
let selected = if crate::bucket::utils::is_meta_bucketname(bucket) && version_id.is_none() {
// A null-version system record must carry the same commit proof
// as GET. Listing only needs the selected metadata, so no disk
// handles are retained from the shared read selector.
Self::select_committed_fileinfo(&vec![None; disk_count], &parts, &errors, "", read_quorum, write_quorum)?.1
} else {
let mod_time = Self::common_time(&Self::list_object_modtimes(&parts, &errors), read_quorum);
Self::pick_valid_fileinfo(&parts, mod_time, None, read_quorum)?
};
let mod_time = Self::common_time(&Self::list_object_modtimes(&parts, &errors), read_quorum);
let selected = Self::pick_valid_fileinfo(&parts, mod_time, None, read_quorum)?;
metadata.add_version(selected)?;
}
if metadata.versions.is_empty() {
+51 -99
View File
@@ -80,7 +80,7 @@ use crate::erasure::codec::bridge::{
use crate::erasure::coding;
use crate::error::{Error, Result, is_err_version_not_found};
use crate::error::{GenericError, ObjectApiError, is_err_object_not_found};
use crate::io_support::bitrot::{create_bitrot_reader_from_bytes, create_bitrot_writer, create_bitrot_writer_with_orig_volume};
use crate::io_support::bitrot::{create_bitrot_reader_from_bytes, create_bitrot_writer};
use crate::object_api::ObjectOptions;
use crate::object_api::get_object_body_cache_hook;
use crate::object_api::object_api_utils::get_raw_etag;
@@ -120,6 +120,7 @@ use bytesize::ByteSize;
use futures::future::join_all;
use http::HeaderMap;
use md5::{Digest as Md5Digest, Md5};
use regex::Regex;
use rustfs_config::MI_B;
use rustfs_filemeta::metadata_keys;
use rustfs_filemeta::{
@@ -153,8 +154,8 @@ use rustfs_utils::http::headers::{
};
use rustfs_utils::http::{
SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE, SUFFIX_BUCKET_INCARNATION_ID, SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE,
SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_RESTORE_OPERATION_ID, SUFFIX_RESTORE_WORKER_LOCK, contains_key_str, get_header_map,
get_str, insert_str, is_object_encryption_marker, remove_header_map,
SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_RESTORE_COPYBACK, SUFFIX_RESTORE_OPERATION_ID, SUFFIX_RESTORE_WORKER_LOCK,
contains_key_str, get_header_map, get_str, insert_str, is_object_encryption_marker, remove_header_map,
};
use rustfs_utils::{
HashAlgorithm,
@@ -8035,29 +8036,16 @@ fn parts_after_marker(part_numbers: &[usize], part_number_marker: usize) -> Opti
Some(&part_numbers[index..])
}
fn canonicalized_etag(etag: &str) -> Cow<'_, str> {
// Preserve the legacy suffix-matching behavior for embedded quotes.
let trimmed = etag.trim_end_matches('"');
let Some(last_quote) = trimmed.rfind('"') else {
return Cow::Borrowed(trimmed);
};
let prefix = trimmed[..last_quote].trim_end_matches('"');
let suffix = &trimmed[last_quote + 1..];
if prefix.is_empty() {
Cow::Borrowed(suffix)
} else {
let mut result = String::with_capacity(prefix.len() + suffix.len());
result.push_str(prefix);
result.push_str(suffix);
Cow::Owned(result)
}
pub fn canonicalize_etag(etag: &str) -> String {
let re = Regex::new("\"*?([^\"]*?)\"*?$").unwrap();
re.replace_all(etag, "$1").to_string()
}
pub fn e_tag_matches(etag: &str, condition: &str) -> bool {
if condition.trim() == "*" {
return true;
}
canonicalized_etag(etag) == canonicalized_etag(condition)
canonicalize_etag(etag) == canonicalize_etag(condition)
}
fn weak_etag_matches(etag: &str, condition: &str) -> bool {
@@ -8072,12 +8060,15 @@ fn weak_etag_matches(etag: &str, condition: &str) -> bool {
.or_else(|| trimmed.strip_prefix("w/"))
.unwrap_or(trimmed)
}
canonicalized_etag(strip_weak(etag)) == canonicalized_etag(strip_weak(condition))
canonicalize_etag(strip_weak(etag)) == canonicalize_etag(strip_weak(condition))
}
pub fn should_prevent_write(oi: &ObjectInfo, if_none_match: Option<&str>, if_match: Option<&str>) -> bool {
let if_none_match = if_none_match.map(str::trim).filter(|condition| !condition.is_empty());
let if_match = if_match.map(str::trim).filter(|condition| !condition.is_empty());
pub fn should_prevent_write(oi: &ObjectInfo, if_none_match: Option<String>, if_match: Option<String>) -> bool {
let if_none_match = if_none_match
.as_deref()
.map(str::trim)
.filter(|condition| !condition.is_empty());
let if_match = if_match.as_deref().map(str::trim).filter(|condition| !condition.is_empty());
match &oi.etag {
Some(etag) => {
@@ -9038,13 +9029,6 @@ mod tests {
"constructing an object lock must not clone each client Arc"
);
// Check shared ownership before acquisition can spawn pending cleanup.
let read_lock = set
.new_ns_lock("bucket", "read-object")
.await
.expect("second namespace lock should be created");
assert_eq!(Arc::strong_count(&set.shared_lockers), clients_before + 2);
let write_error = write_lock
.get_write_lock(Duration::from_millis(500))
.await
@@ -9059,6 +9043,11 @@ mod tests {
),
"the shared client representation must preserve the exact write quorum result: {write_error}"
);
let read_lock = set
.new_ns_lock("bucket", "read-object")
.await
.expect("second namespace lock should be created");
assert_eq!(Arc::strong_count(&set.shared_lockers), clients_before + 2);
let read_guard = read_lock
.get_read_lock(Duration::from_millis(500))
.await
@@ -13783,22 +13772,6 @@ mod tests {
assert!(result2.iter().all(|d| d.is_none()));
}
#[test]
fn canonicalized_etag_preserves_legacy_regex_semantics() {
let regex = regex::Regex::new("\"*?([^\"]*?)\"*?$").expect("valid legacy ETag pattern");
let alphabet = ['a', '"', '\n', 'é', '\0'];
for length in 0..=6_u32 {
for mut number in 0..alphabet.len().pow(length) {
let mut input = String::new();
for _ in 0..length {
input.push(alphabet[number % alphabet.len()]);
number /= alphabet.len();
}
assert_eq!(canonicalized_etag(&input), regex.replace_all(&input, "$1"), "input: {input:?}");
}
}
}
#[test]
fn test_etag_matches() {
assert!(e_tag_matches("abc", "abc"));
@@ -14007,10 +13980,7 @@ mod tests {
let requested_until = now + Duration::from_secs(60 * 60 * 24);
let mut user_defined = HashMap::new();
user_defined.insert(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(),
);
user_defined.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string());
user_defined.insert(
metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(),
existing_until.format(&time::format_description::well_known::Rfc3339).unwrap(),
@@ -14022,7 +13992,7 @@ mod tests {
};
let opts = ObjectOptions {
object_lock_retention: Some(crate::storage_api_contracts::object::ObjectLockRetentionOptions {
mode: Some(s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string()),
mode: Some("COMPLIANCE".to_string()),
retain_until: Some(requested_until),
bypass_governance: true,
}),
@@ -14042,10 +14012,7 @@ mod tests {
let requested_until = now + Duration::from_secs(60 * 60 * 24);
let mut user_defined = HashMap::new();
user_defined.insert(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
s3s::dto::ObjectLockRetentionMode::GOVERNANCE.to_string(),
);
user_defined.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), "GOVERNANCE".to_string());
user_defined.insert(
metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(),
existing_until.format(&time::format_description::well_known::Rfc3339).unwrap(),
@@ -14057,7 +14024,7 @@ mod tests {
};
let opts = ObjectOptions {
object_lock_retention: Some(crate::storage_api_contracts::object::ObjectLockRetentionOptions {
mode: Some(s3s::dto::ObjectLockRetentionMode::GOVERNANCE.to_string()),
mode: Some("GOVERNANCE".to_string()),
retain_until: Some(requested_until),
bypass_governance: true,
}),
@@ -14072,10 +14039,7 @@ mod tests {
async fn test_check_object_lock_delete_blocks_compliance_version_delete() {
let retain_until = OffsetDateTime::now_utc() + Duration::from_secs(60 * 60 * 24 * 60);
let mut user_defined = HashMap::new();
user_defined.insert(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(),
);
user_defined.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string());
user_defined.insert(
metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(),
retain_until.format(&time::format_description::well_known::Rfc3339).unwrap(),
@@ -14117,10 +14081,7 @@ mod tests {
},
restore_expires: Some(restore_expiry),
user_defined: Arc::new(HashMap::from([
(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(),
),
(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()),
(
metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(),
retain_until.format(&time::format_description::well_known::Rfc3339).unwrap(),
@@ -14191,10 +14152,7 @@ mod tests {
async fn test_check_object_lock_delete_allows_versioned_delete_marker_creation() {
let retain_until = OffsetDateTime::now_utc() + Duration::from_secs(60 * 60 * 24 * 60);
let mut user_defined = HashMap::new();
user_defined.insert(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(),
);
user_defined.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string());
user_defined.insert(
metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(),
retain_until.format(&time::format_description::well_known::Rfc3339).unwrap(),
@@ -14219,10 +14177,7 @@ mod tests {
fn governance_retained_obj_info() -> ObjectInfo {
let retain_until = OffsetDateTime::now_utc() + Duration::from_secs(60 * 60 * 24 * 60);
let mut user_defined = HashMap::new();
user_defined.insert(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
s3s::dto::ObjectLockRetentionMode::GOVERNANCE.to_string(),
);
user_defined.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), "GOVERNANCE".to_string());
user_defined.insert(
metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(),
retain_until.format(&time::format_description::well_known::Rfc3339).unwrap(),
@@ -14272,10 +14227,7 @@ mod tests {
async fn test_check_object_lock_delete_blocks_replicated_compliance_version_purge() {
let retain_until = OffsetDateTime::now_utc() + Duration::from_secs(60 * 60 * 24 * 60);
let mut user_defined = HashMap::new();
user_defined.insert(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(),
);
user_defined.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string());
user_defined.insert(
metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(),
retain_until.format(&time::format_description::well_known::Rfc3339).unwrap(),
@@ -14330,11 +14282,11 @@ mod tests {
#[test]
fn test_object_lock_delete_check_required_keeps_object_lock_config_buckets() {
use s3s::dto::{ObjectLockConfiguration, ObjectLockEnabled};
use rustfs_gateway_types::persistence::PersistedObjectLockConfiguration as ObjectLockConfiguration;
let mut bm = crate::bucket::metadata::BucketMetadata::new("lock-config-bucket");
bm.object_lock_config = Some(ObjectLockConfiguration {
object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
object_lock_enabled: Some("Enabled".to_owned()),
..Default::default()
});
assert!(object_lock_delete_check_required(Some(&bm)));
@@ -14374,61 +14326,61 @@ mod tests {
etag: Some("abc".to_string()),
..Default::default()
};
let if_none_match = Some("abc");
let if_none_match = Some("abc".to_string());
let if_match = None;
assert!(should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = Some("*");
let if_none_match = Some("*".to_string());
let if_match = None;
assert!(should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = None;
let if_match = Some("def");
let if_match = Some("def".to_string());
assert!(should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = None;
let if_match = Some("*");
let if_match = Some("*".to_string());
assert!(!should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = Some("def");
let if_none_match = Some("def".to_string());
let if_match = None;
assert!(!should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = Some("def");
let if_match = Some("*");
let if_none_match = Some("def".to_string());
let if_match = Some("*".to_string());
assert!(!should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = Some("def");
let if_match = Some("\"abc\"");
let if_none_match = Some("def".to_string());
let if_match = Some("\"abc\"".to_string());
assert!(!should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = Some("*");
let if_match = Some("\"abc\"");
let if_none_match = Some("*".to_string());
let if_match = Some("\"abc\"".to_string());
assert!(should_prevent_write(&oi, if_none_match, if_match));
let oi = ObjectInfo {
etag: None,
..Default::default()
};
let if_none_match = Some("*");
let if_match = Some("\"abc\"");
let if_none_match = Some("*".to_string());
let if_match = Some("\"abc\"".to_string());
assert!(should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = None;
let if_match = None;
assert!(!should_prevent_write(&oi, if_none_match, if_match));
let if_none_match = Some("");
let if_match = Some(" ");
let if_none_match = Some(String::new());
let if_match = Some(" ".to_string());
assert!(!should_prevent_write(&oi, if_none_match, if_match));
let oi = ObjectInfo {
etag: Some("abc".to_string()),
..Default::default()
};
assert!(should_prevent_write(&oi, Some("W/\"abc\""), None));
assert!(!should_prevent_write(&oi, Some("W/\"def\""), None));
assert!(should_prevent_write(&oi, None, Some("W/\"abc\"")));
assert!(should_prevent_write(&oi, Some("W/\"abc\"".to_string()), None));
assert!(!should_prevent_write(&oi, Some("W/\"def\"".to_string()), None));
assert!(should_prevent_write(&oi, None, Some("W/\"abc\"".to_string())));
}
#[test]
@@ -17231,8 +17183,8 @@ mod tests {
// otherwise `from_file_info` never synthesizes the null version id and
// the branch under test is not reached.
let suspended = crate::bucket::replication::DeleteReplicationConfigSnapshot::from_configs_for_test(
s3s::dto::VersioningConfiguration {
status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::SUSPENDED)),
rustfs_gateway_types::persistence::PersistedVersioningConfiguration {
status: Some("Suspended".to_owned()),
..Default::default()
},
None,
@@ -2374,6 +2374,86 @@ mod tests {
}
}
#[tokio::test]
async fn delete_recovery_batch_with_missing_version_recovers_after_preparation_failure() {
use crate::disk::RUSTFS_META_TMP_BUCKET;
use crate::storage_api_contracts::object::ObjectToDelete;
let bucket = "recover-batch-missing-version";
let (dirs, store, disks) = fixture(bucket).await;
let deleted = put(&store, bucket, &vec![48; 4096], true).await.expect("version to delete");
let remaining = vec![49; 4096];
let retained = put(&store, bucket, &remaining, true).await.expect("version to retain");
let missing = Uuid::new_v4();
let temporary = dirs[3].path().join(RUSTFS_META_TMP_BUCKET);
let saved = dirs[3].path().join("saved-delete-tmp");
tokio::fs::rename(&temporary, &saved)
.await
.expect("save returning disk's staging directory");
tokio::fs::write(&temporary, b"block preparation")
.await
.expect("fail staging after the recovery preflight");
let (objects, errors) = store
.delete_objects(
bucket,
[deleted, missing]
.into_iter()
.map(|version| ObjectToDelete {
object_name: "object".into(),
version_id: Some(version),
..Default::default()
})
.collect(),
ObjectOptions {
versioned: true,
..Default::default()
},
)
.await;
assert_eq!(objects.len(), 2);
assert!(errors.iter().all(Option::is_none), "batch deletion succeeds at quorum: {errors:?}");
assert!(
disks[3]
.discover_delete_transactions(bucket, Some("object"), None, 100)
.await
.expect("inspect failed preparation")
.transactions
.is_empty(),
"returning disk must adopt the quorum decision without a local preparation"
);
tokio::fs::remove_file(&temporary).await.expect("remove preparation failure");
tokio::fs::rename(&saved, &temporary)
.await
.expect("restore returning disk staging");
let result = store
.recover_delete_transactions(bucket, &HealOpts::default())
.await
.expect("recover batch containing a missing non-null version");
assert!(result.detail.contains("blocked=0"), "{}", result.detail);
assert_eq!(body(&store, bucket, Some(retained)).await, remaining);
for version in [deleted, missing] {
assert_eq!(
disks[3]
.read_version(bucket, bucket, "object", &version.to_string(), &ReadOptions::default())
.await
.expect_err("returning disk must not retain or create a deleted version"),
DiskError::FileVersionNotFound
);
}
for disk in &disks {
assert!(
disk.discover_delete_transactions(bucket, Some("object"), None, 100)
.await
.expect("inspect converged batch")
.transactions
.is_empty(),
"every participant must converge and release the decision"
);
}
}
#[tokio::test]
async fn delete_recovery_dry_run_matches_adoption_of_already_absent_non_null_version() {
let bucket = "recover-preview-non-null";
@@ -2740,6 +2820,140 @@ mod tests {
.expect("no retained version decision may keep the empty bucket nonempty");
}
#[tokio::test]
async fn delete_recovery_partial_batch_receipt_handles_missing_versions() {
use crate::disk::RUSTFS_META_TMP_BUCKET;
use crate::set_disk::ops::object::DeleteCleanupBarrier;
use crate::storage_api_contracts::object::ObjectToDelete;
for include_missing in [false, true] {
let bucket = "recover-prepared-batch-missing-versions";
let (dirs, store, disks) = fixture(bucket).await;
metadata_sys::update_in(
&store.ctx,
bucket,
crate::bucket::metadata::BUCKET_VERSIONING_CONFIG,
b"<VersioningConfiguration xmlns=\"http://s3.amazonaws.com/doc/2006-03-01/\"><Status>Enabled</Status></VersioningConfiguration>".to_vec(),
)
.await
.expect("enable supported versioned bucket");
let first = put(&store, bucket, &vec![81; 1024 * 1024], true)
.await
.expect("first version");
let second = put(&store, bucket, &vec![82; 4096], true).await.expect("retained version");
let missing = Uuid::new_v4();
let barrier = DeleteCleanupBarrier::install(bucket);
let deleting = store.clone();
let task = tokio::spawn(async move {
let mut versions = vec![first];
if include_missing {
versions.push(missing);
}
deleting
.delete_objects(
bucket,
versions
.into_iter()
.map(|version| ObjectToDelete {
object_name: "object".into(),
version_id: Some(version),
..Default::default()
})
.collect(),
ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
});
barrier.wait_until_paused().await;
let transactions = disks[0]
.discover_delete_transactions(bucket, Some("object"), None, 100)
.await
.expect("discover batch preparation")
.transactions;
assert_eq!(transactions.len(), 1);
let transaction = transactions[0].transaction;
for disk in &disks {
let state = disk
.inspect_delete_transaction(bucket, "object", transaction)
.await
.expect("inspect prepared batch");
assert_eq!(state.state, DeleteRecoveryState::NewPrepared);
let journal = state.journal.expect("prepared journal");
assert!(journal.original_digest.is_some());
assert!(journal.original_versions.iter().all(|v| v.version_id != Some(missing)));
}
// Fail durable receipt staging on two participants after metadata quorum.
for dir in dirs.iter().skip(2) {
tokio::fs::rename(dir.path().join(RUSTFS_META_TMP_BUCKET), dir.path().join("saved-receipt-staging"))
.await
.expect("save receipt staging directory");
tokio::fs::write(dir.path().join(RUSTFS_META_TMP_BUCKET), b"receipt staging I/O failure")
.await
.expect("inject receipt staging failure");
}
barrier.release();
let (_, errors) = task.await.expect("batch deletion must join");
drop(barrier);
assert_eq!(errors.len(), if include_missing { 2 } else { 1 });
assert!(errors.iter().all(Option::is_some), "partial receipts are indeterminate: {errors:?}");
let mut receipts = 0;
let mut prepared = 0;
for disk in &disks {
match disk
.inspect_delete_transaction(bucket, "object", transaction)
.await
.expect("partial decision")
.state
{
DeleteRecoveryState::QuorumCommitted => receipts += 1,
DeleteRecoveryState::NewPrepared => prepared += 1,
state => panic!("unexpected post-publication state: {state:?}"),
}
}
assert_eq!((receipts, prepared), (2, 2));
for dir in dirs.iter().skip(2) {
tokio::fs::remove_file(dir.path().join(RUSTFS_META_TMP_BUCKET))
.await
.expect("remove injected staging failure");
tokio::fs::rename(dir.path().join("saved-receipt-staging"), dir.path().join(RUSTFS_META_TMP_BUCKET))
.await
.expect("restore receipt staging");
}
store
.delete_object(
bucket,
"object",
ObjectOptions {
versioned: true,
version_id: Some(second.to_string()),
..Default::default()
},
)
.await
.expect("later version deletion succeeds");
store
.recover_delete_transactions(bucket, &HealOpts::default())
.await
.expect("certified batch must recover after later version deletion");
for disk in &disks {
assert!(
disk.discover_delete_transactions(bucket, Some("object"), None, 100)
.await
.expect("settled transaction inventory")
.transactions
.is_empty()
);
}
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("settled receipts must not block removal of the empty bucket");
}
}
#[tokio::test]
async fn delete_recovery_partial_receipt_converges_after_later_version_absence() {
let bucket = "recover-partial-chained-versions";
+35 -256
View File
@@ -38,12 +38,12 @@ use super::super::{
ObjectLockDiagGuard, ObjectOptions, ObjectPartInfo, OffsetDateTime, PartInfo, PutObjReader, RUSTFS_META_MULTIPART_BUCKET,
RUSTFS_META_TMP_BUCKET, RUSTFS_MULTIPART_BUCKET_KEY, RUSTFS_MULTIPART_OBJECT_KEY, Result, SLASH_SEPARATOR,
SUFFIX_ACTUAL_OBJECT_SIZE_CAP, SUFFIX_ACTUAL_SIZE, SUFFIX_BUCKET_INCARNATION_ID, SUFFIX_COMPRESSION_SIZE,
SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_RESTORE_OPERATION_ID, SUFFIX_RESTORE_WORKER_LOCK, SetDisks, SmallWritePath, StorageError,
Uuid, WriteLayout, check_object_lock_for_deletion_with_state, classify_multipart_part_write_path, coding,
complete_multipart_part_error, complete_multipart_part_error_result, complete_part_checksum, contains_key_str,
create_bitrot_writer_with_orig_volume, debug, disk, error, file_info_is_valid_for_metadata, get_complete_multipart_md5,
get_header_map, get_str, insert_str, is_err_object_not_found, is_err_version_not_found, is_min_allowed_part_size,
log_multipart_write_quorum_failure, multipart_part_logical_size, parts_after_marker, path_join_buf,
SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_RESTORE_COPYBACK, SUFFIX_RESTORE_OPERATION_ID, SUFFIX_RESTORE_WORKER_LOCK, SetDisks,
SmallWritePath, StorageError, Uuid, WriteLayout, check_object_lock_for_deletion_with_state,
classify_multipart_part_write_path, coding, complete_multipart_part_error, complete_multipart_part_error_result,
complete_part_checksum, contains_key_str, create_bitrot_writer, debug, disk, error, file_info_is_valid_for_metadata,
get_complete_multipart_md5, get_header_map, get_str, insert_str, is_err_object_not_found, is_err_version_not_found,
is_min_allowed_part_size, log_multipart_write_quorum_failure, multipart_part_logical_size, parts_after_marker, path_join_buf,
prepare_complete_multipart_parts, record_compression_total_memory, record_multipart_write_quorum_failure,
reduce_read_quorum_errs, reduce_write_quorum_errs, remove_header_map, resolve_write_layout,
restore_commit_operation_id_from_metadata, should_persist_encryption_original_size, strip_internal_multipart_metadata,
@@ -124,10 +124,9 @@ async fn create_part_writers(
let Some(disk) = disk else {
return (None, Some(DiskError::DiskNotFound));
};
match create_bitrot_writer_with_orig_volume(
match create_bitrot_writer(
false,
Some(disk),
RUSTFS_META_MULTIPART_BUCKET,
RUSTFS_META_TMP_BUCKET,
path,
length,
@@ -978,7 +977,7 @@ async fn multipart_upload_paths_on_disk(disk: DiskStore, bucket: &str, root_pref
return Err(DiskError::DiskNotFound);
}
if !root_prefix.is_empty() && root_prefix != DATA_MOVEMENT_MULTIPART_PREFIX {
if !root_prefix.is_empty() {
let upload_dirs = match disk.list_dir(bucket, RUSTFS_META_MULTIPART_BUCKET, root_prefix, -1).await {
Ok(entries) => entries,
Err(DiskError::FileNotFound | DiskError::VolumeNotFound) => return Ok(Vec::new()),
@@ -1003,13 +1002,6 @@ async fn multipart_upload_paths_on_disk(disk: DiskStore, bucket: &str, root_pref
.into_iter()
.map(|sha_dir| sha_dir.trim_end_matches('/').to_owned())
.filter(|sha_dir| sha_dir.len() == 64 && sha_dir.bytes().all(|byte| byte.is_ascii_hexdigit()))
.map(|sha_dir| {
if root_prefix.is_empty() {
sha_dir
} else {
format!("{root_prefix}/{sha_dir}")
}
})
.collect::<Vec<_>>();
let listed_upload_dirs = stream::iter(sha_dirs)
.map(|sha_dir| {
@@ -1132,7 +1124,6 @@ impl SetDisks {
orig_bucket: &str,
error_path: &str,
root_prefix: &str,
require_all_disks: bool,
) -> Result<(Vec<Option<DiskStore>>, Vec<String>, usize, bool)> {
let disks = self.disks.read().await.clone();
if disks.is_empty() {
@@ -1174,8 +1165,7 @@ impl SetDisks {
}
}
let required_discoveries = if require_all_disks { disks.len() } else { discovery_quorum };
if let Some(err) = reduce_read_quorum_errs(&discovery_errors, OBJECT_OP_IGNORED_ERRS, required_discoveries) {
if let Some(err) = reduce_read_quorum_errs(&discovery_errors, OBJECT_OP_IGNORED_ERRS, discovery_quorum) {
return Err(to_object_err(err.into(), vec![orig_bucket, error_path]));
}
@@ -1197,7 +1187,7 @@ impl SetDisks {
pub(crate) async fn first_multipart_upload_path_for_decommission(&self, bucket: &str) -> Result<Option<String>> {
let (_, paths, _, _) = self
.discover_multipart_upload_paths(bucket, RUSTFS_META_MULTIPART_BUCKET, "", false)
.discover_multipart_upload_paths(bucket, RUSTFS_META_MULTIPART_BUCKET, "")
.await?;
Ok(paths.into_iter().next())
}
@@ -1210,9 +1200,8 @@ impl SetDisks {
upload_identity: &str,
) -> Result<Vec<String>> {
let expected_parent = format!("{DATA_MOVEMENT_MULTIPART_PREFIX}/{}", Self::get_multipart_sha_dir(bucket, object));
let (_, candidate_paths, _, has_minority_candidate) = self
.discover_multipart_upload_paths(bucket, object, &expected_parent, false)
.await?;
let (_, candidate_paths, _, has_minority_candidate) =
self.discover_multipart_upload_paths(bucket, object, &expected_parent).await?;
if has_minority_candidate {
return Err(Error::DecommissionCapacityBlocked {
message: "data movement multipart cleanup found an upload path on fewer than the discovery quorum of disks"
@@ -1761,54 +1750,6 @@ impl SetDisks {
})
}
pub(crate) async fn ensure_no_multipart_uploads(&self, bucket: &str) -> Result<()> {
// Every disk must confirm absence in both namespaces before cleanup can
// remove catalog protection from the bucket.
for root_prefix in ["", DATA_MOVEMENT_MULTIPART_PREFIX] {
let (disks, paths, _, has_minority_candidate) =
self.discover_multipart_upload_paths(bucket, "", root_prefix, true).await?;
if has_minority_candidate {
return Err(Error::ErasureReadQuorum);
}
for upload_path in paths {
let (sha_dir, _) = upload_path
.rsplit_once('/')
.filter(|(directory, upload_id)| !directory.is_empty() && !upload_id.is_empty())
.ok_or(DiskError::CorruptedFormat)?;
let (metadata, errors) =
Self::read_all_fileinfo(&disks, bucket, RUSTFS_META_MULTIPART_BUCKET, &upload_path, "", false, false, false)
.await?;
let (read_quorum, _) = Self::object_quorum_from_meta(&metadata, &errors, self.default_parity_count)?;
let read_quorum = usize::try_from(read_quorum).map_err(|_| DiskError::ErasureReadQuorum)?;
if let Some(error) = reduce_read_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, read_quorum) {
return Err(to_object_err(error.into(), vec![bucket, &upload_path]));
}
let (_, modified, etag) = Self::list_online_disks(&disks, &metadata, &errors, read_quorum);
let file = Self::pick_valid_fileinfo(&metadata, modified, etag, read_quorum)?;
let (Some(owner_bucket), Some(owner_object)) = (
file.metadata.get(RUSTFS_MULTIPART_BUCKET_KEY),
file.metadata.get(RUSTFS_MULTIPART_OBJECT_KEY),
) else {
return Err(to_object_err(DiskError::CorruptedFormat.into(), vec![bucket, &upload_path]));
};
if owner_bucket.is_empty()
|| owner_object.is_empty()
|| Self::get_multipart_sha_dir(owner_bucket, owner_object)
!= sha_dir
.strip_prefix(DATA_MOVEMENT_MULTIPART_PREFIX)
.and_then(|suffix| suffix.strip_prefix('/'))
.unwrap_or(sha_dir)
{
return Err(to_object_err(DiskError::CorruptedFormat.into(), vec![bucket, &upload_path]));
}
if owner_bucket == bucket {
return Err(Error::BucketNotEmpty(bucket.to_string()));
}
}
}
Ok(())
}
#[allow(clippy::too_many_arguments)]
pub(crate) async fn list_multipart_uploads_for_incarnation(
&self,
@@ -1820,8 +1761,7 @@ impl SetDisks {
max_uploads: usize,
expected_incarnation_id: Option<Uuid>,
) -> Result<ListMultipartsInfo> {
let (disks, candidate_paths, discovery_quorum, _) =
self.discover_multipart_upload_paths(bucket, prefix, "", false).await?;
let (disks, candidate_paths, discovery_quorum, _) = self.discover_multipart_upload_paths(bucket, prefix, "").await?;
let listed_uploads = stream::iter(candidate_paths)
.map(|upload_path| {
let disks = &disks;
@@ -2496,16 +2436,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
}
let part_integrity = if let Some(integrity) = integrity {
Some(
integrity
.write(
&mut shuffle_disks,
RUSTFS_META_MULTIPART_BUCKET,
RUSTFS_META_TMP_BUCKET,
&tmp_part,
)
.await?,
)
Some(integrity.write(&mut shuffle_disks, bucket, RUSTFS_META_TMP_BUCKET, &tmp_part).await?)
} else { None };
if shuffle_disks.iter().filter(|disk| disk.is_some()).count() < write_quorum {
return Err(Error::ErasureWriteQuorum);
@@ -3502,7 +3433,14 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
crate::object_api::ENCRYPTED_PART_LAYOUT_CANDIDATE_SUFFIX,
);
rustfs_utils::http::metadata_compat::remove_str(&mut fi.metadata, crate::object_api::ENCRYPTED_PART_LAYOUT_QUORUM_SUFFIX);
if expected_restore_operation_id.is_some() {
// Pending copy-back keeps its recovery identity until finalization.
if expected_restore_operation_id.is_some()
&& !fi
.metadata
.get(metadata_keys::RESTORE)
.and_then(|value| rustfs_filemeta::parse_restore_obj_status(value).ok())
.is_some_and(|status| status.is_restore_in_progress == Some(true))
{
rustfs_utils::http::metadata_compat::remove_str(&mut fi.metadata, SUFFIX_RESTORE_OPERATION_ID);
rustfs_utils::http::metadata_compat::remove_str(&mut fi.metadata, SUFFIX_RESTORE_WORKER_LOCK);
}
@@ -3540,6 +3478,10 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
bind_multipart_proof_snapshot(&disks, &self.multipart_proof_verification_context())?;
let part_path = format!("{}/{}/", upload_id_path, fi.data_dir.unwrap_or(Uuid::nil()));
self.recover_part_transactions(&part_path, read_quorum, write_quorum)
.await
.map_err(|err| to_object_err(err.into(), vec![bucket, object]))?;
let part_meta_paths = uploaded_parts
.iter()
.map(|v| format!("{part_path}part.{0}.meta", v.part_num))
@@ -3547,10 +3489,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
let part_numbers = uploaded_parts.iter().map(|v| v.part_num).collect::<Vec<usize>>();
self.recover_part_transactions(&part_path, read_quorum, write_quorum)
.await
.map_err(|err| to_object_err(err.into(), vec![bucket, object]))?;
let object_parts = Self::read_parts(&disks, RUSTFS_META_MULTIPART_BUCKET, &part_meta_paths, &part_numbers, read_quorum)
.await
.map_err(|err| to_object_err(err.into(), vec![bucket, object]))?;
@@ -3887,6 +3825,13 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
fi.metadata.remove(rustfs_rio::RUSTFS_MULTIPART_CHECKSUM);
fi.metadata.remove(rustfs_rio::RUSTFS_MULTIPART_CHECKSUM_TYPE);
strip_internal_multipart_metadata(&mut fi.metadata);
if opts.restore_copyback {
insert_str(
&mut fi.metadata,
SUFFIX_RESTORE_COPYBACK,
rustfs_utils::http::metadata_compat::RESTORE_COPYBACK_PROTOCOL_V1.to_string(),
);
}
// A previous coordinator can stop after a minority publication without
// undo. A fresh final directory keeps this attempt separate from old
@@ -4458,7 +4403,6 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
let needs_immediate_heal = rename_commit.needs_immediate_heal();
let op_old_dir = rename_commit.data_dir;
let cleanup_disks = rename_commit.cleanup_disks;
let old_current_size = rename_commit.old_current_size;
let committed_file_info = rename_commit.committed_file_info;
if needs_immediate_heal {
@@ -4527,13 +4471,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
// parts; deleting them before the commit would strand the upload
// permanently. This mirrors the "clean up only after commit" pattern
// already used for the old data-dir GC and the upload-dir delete_all below.
let can_skip_part_cleanup = !needs_upgrade
&& commit_disks
.iter()
.all(|disk| disk.as_ref().is_some_and(|disk| disk.is_local()));
if !can_skip_part_cleanup {
commit_set.cleanup_multipart_path(&parts).await;
}
commit_set.cleanup_multipart_path(&parts).await;
if let Some(old_dir) = op_old_dir {
// backlog#898: best-effort reclaim of the dereferenced old data dir.
@@ -4579,9 +4517,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
drop(_upload_guard.take());
let mut obj_info = ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned);
obj_info.backfilled_old_current_size = old_current_size;
Ok(obj_info)
Ok(ObjectInfo::from_file_info(&fi, &commit_bucket, &commit_object, commit_is_versioned))
};
let result = if detach_commit_owner {
@@ -11272,163 +11208,6 @@ mod tests {
assert_eq!(upload_uuid_suffix(&recovered[0]), upload_uuid_suffix(&upload.upload_id));
}
#[tokio::test]
async fn multipart_absence_inspection_distinguishes_bucket_ownership() {
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "multipart-cleanup-target";
let other = "multipart-cleanup-other";
for disk in &disk_stores {
disk.make_volume(bucket).await.expect("create target bucket");
disk.make_volume(other).await.expect("create other bucket");
}
set_disks
.ensure_no_multipart_uploads(bucket)
.await
.expect("an empty upload namespace permits cleanup");
set_disks
.new_multipart_upload(other, "object.bin", &ObjectOptions::default())
.await
.expect("create upload in other bucket");
set_disks
.ensure_no_multipart_uploads(bucket)
.await
.expect("an upload owned by another bucket permits cleanup");
set_disks
.new_multipart_upload(bucket, "object.bin", &ObjectOptions::default())
.await
.expect("create upload in target bucket");
assert!(
matches!(set_disks.ensure_no_multipart_uploads(bucket).await, Err(Error::BucketNotEmpty(name)) if name == bucket)
);
}
#[tokio::test]
async fn multipart_absence_inspection_requires_every_disk_online() {
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "multipart-cleanup-offline";
for disk in &disk_stores {
disk.make_volume(bucket).await.expect("create bucket");
}
set_disks.disks.write().await[3] = None;
let listed = set_disks
.list_multipart_uploads_for_incarnation(bucket, "", None, None, None, 1000, None)
.await
.expect("ordinary listing preserves its discovery quorum");
assert!(listed.uploads.is_empty());
let error = set_disks
.ensure_no_multipart_uploads(bucket)
.await
.expect_err("an offline disk must prevent confirmation of an empty multipart namespace");
assert!(
matches!(&error, Error::InsufficientReadQuorum(error_bucket, error_path) if error_bucket == bucket && error_path.is_empty()),
"the offline disk must produce a bucket-scoped read quorum error: {error:?}"
);
set_disks.disks.write().await[3] = Some(disk_stores[3].clone());
set_disks
.ensure_no_multipart_uploads(bucket)
.await
.expect("all disks can confirm the empty namespace after recovery");
}
#[tokio::test]
async fn multipart_absence_inspection_rejects_a_failed_disk_enumeration() {
let (temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "multipart-cleanup-list-error";
for disk in &disk_stores {
disk.make_volume(bucket).await.expect("create bucket");
}
let sha_dir = SetDisks::get_multipart_sha_dir(bucket, "object.bin");
let multipart_dir = temp_dirs[0].path().join(RUSTFS_META_MULTIPART_BUCKET);
tokio::fs::create_dir_all(&multipart_dir)
.await
.expect("create multipart staging directory");
tokio::fs::write(multipart_dir.join(&sha_dir), b"not a directory")
.await
.expect("replace one upload parent directory with a file");
multipart_upload_paths_on_disk(disk_stores[0].clone(), bucket, "")
.await
.expect_err("the affected disk must fail directory enumeration");
let listed = set_disks
.list_multipart_uploads_for_incarnation(bucket, "", None, None, None, 1000, None)
.await
.expect("ordinary listing still accepts the remaining discovery quorum");
assert!(listed.uploads.is_empty());
let error = set_disks
.ensure_no_multipart_uploads(bucket)
.await
.expect_err("a failed disk enumeration must prevent confirmation of an empty multipart namespace");
assert!(
matches!(&error, Error::InsufficientReadQuorum(error_bucket, error_path) if error_bucket == bucket && error_path.is_empty()),
"the failed enumeration must produce a bucket-scoped read quorum error: {error:?}"
);
tokio::fs::remove_file(temp_dirs[0].path().join(RUSTFS_META_MULTIPART_BUCKET).join(sha_dir))
.await
.expect("remove the invalid directory entry");
set_disks
.ensure_no_multipart_uploads(bucket)
.await
.expect("all disks can confirm the empty namespace after repair");
}
#[tokio::test]
async fn multipart_absence_inspection_rejects_data_movement_uploads() {
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "multipart-cleanup-data-movement";
for disk in &disk_stores {
disk.make_volume(bucket).await.expect("create bucket");
}
let mut metadata = HashMap::new();
rustfs_utils::http::insert_str(
&mut metadata,
rustfs_utils::http::SUFFIX_DATA_MOVEMENT_UPLOAD,
format!("v1:{}:{}", Uuid::new_v4(), OffsetDateTime::now_utc().unix_timestamp_nanos()),
);
set_disks
.new_multipart_upload(
bucket,
"object.bin",
&ObjectOptions {
data_movement: true,
user_defined: metadata,
..Default::default()
},
)
.await
.expect("create data movement upload");
assert!(
matches!(set_disks.ensure_no_multipart_uploads(bucket).await, Err(Error::BucketNotEmpty(name)) if name == bucket)
);
}
#[tokio::test]
async fn multipart_absence_inspection_rejects_unreadable_ownership() {
let (temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "multipart-cleanup-corrupt";
for disk in &disk_stores {
disk.make_volume(bucket).await.expect("create bucket");
}
set_disks
.new_multipart_upload(bucket, "object.bin", &ObjectOptions::default())
.await
.expect("create multipart upload");
let mut corrupted = 0;
for temp_dir in &temp_dirs {
for path in multipart_meta_files_on_disk(temp_dir, "xl.meta").await {
tokio::fs::write(path, b"invalid upload metadata")
.await
.expect("corrupt upload metadata");
corrupted += 1;
}
}
assert!(corrupted > 0, "upload metadata must exist before corruption");
set_disks
.ensure_no_multipart_uploads(bucket)
.await
.expect_err("unreadable upload ownership must block cleanup");
}
/// Regression (issue #5716): a single upload directory whose `xl.meta` was
/// destroyed (crash mid-write, torn disk state) must degrade to that upload
/// alone. Failing the whole ListMultipartUploads turns one piece of stale
+229 -147
View File
@@ -41,23 +41,24 @@ use super::super::{
ObjectKey, ObjectLockConfigSnapshot, ObjectLockConfigState, ObjectOptions, ObjectReader, ObjectToDelete, OffsetDateTime,
Ordering, Pin, PutObjReader, RUSTFS_META_BUCKET, RUSTFS_META_TMP_BUCKET, ReadPathPlan, ReaderImpl, ReplicateDecision,
ReplicationObjectBridge, Result, SET_DISK_COMMIT_TAIL_WARN_THRESHOLD_MS, SLASH_SEPARATOR, SUFFIX_ACTUAL_SIZE,
SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_RESTORE_OPERATION_ID, SUFFIX_RESTORE_WORKER_LOCK, SetDisks,
SmallWritePath, StorageError, TRANSITION_COMPLETE, UpdateMetadataOpts, Uuid, WriteLayout, adaptive_duplex_buffer_size,
build_get_object_info, build_inline_bitrot_readers, build_inline_bitrot_readers_from_refs, can_try_inline_data_shards_direct,
check_object_lock_delete, check_object_lock_for_deletion_with_state, check_object_lock_retention_update,
classify_get_codec_streaming_object_class, classify_put_write_path, classify_storage_error,
collect_inline_data_shard_fileinfos_by_index, contains_key_str, create_bitrot_writer_with_orig_volume, debug,
delete_file_info_version_id, disk, ensure_delete_commit_locks_held, error, explicit_delete_removed_marker,
finish_set_disk_read_lock, get_codec_streaming_reader_gate_with_plan, get_object_body_cache_hook, get_raw_etag,
get_small_object_direct_memory_decision_with_threshold_and_plan, get_small_object_direct_memory_threshold,
get_stage_timer_if_enabled, get_str, get_transitioned_object_reader_with_tier_manager, inline_admission_shard_size,
inline_erasure_shard_file_offset, inline_erasure_shard_size, insert_str, is_deadlock_detection_enabled,
is_err_object_not_found, is_err_version_not_found, is_explicit_null_version, is_get_codec_streaming_base_enabled,
is_get_small_object_direct_memory_enabled, is_lock_optimization_enabled, issue3031_diag_enabled, join_all,
known_put_object_storage_size, path_join_buf, put_restore_opts, record_compression_total_memory,
record_get_codec_streaming_gate_decision, record_get_direct_memory_decision, record_get_object_pipeline_failure,
record_get_object_pipeline_failure_for_path, record_get_object_reader_path_observation, record_get_stage_duration_if_enabled,
record_lock_acquire, reduce_write_quorum_errs, release_materialized_read_lock, replication_write_may_pass_worm_gate,
SUFFIX_COMPRESSION, SUFFIX_COMPRESSION_SIZE, SUFFIX_RESTORE_COPYBACK, SUFFIX_RESTORE_OPERATION_ID,
SUFFIX_RESTORE_WORKER_LOCK, SetDisks, SmallWritePath, StorageError, TRANSITION_COMPLETE, UpdateMetadataOpts, Uuid,
WriteLayout, adaptive_duplex_buffer_size, build_get_object_info, build_inline_bitrot_readers,
build_inline_bitrot_readers_from_refs, can_try_inline_data_shards_direct, check_object_lock_delete,
check_object_lock_for_deletion_with_state, check_object_lock_retention_update, classify_get_codec_streaming_object_class,
classify_put_write_path, classify_storage_error, collect_inline_data_shard_fileinfos_by_index, contains_key_str,
create_bitrot_writer, debug, delete_file_info_version_id, disk, ensure_delete_commit_locks_held, error,
explicit_delete_removed_marker, finish_set_disk_read_lock, get_codec_streaming_reader_gate_with_plan,
get_object_body_cache_hook, get_raw_etag, get_small_object_direct_memory_decision_with_threshold_and_plan,
get_small_object_direct_memory_threshold, get_stage_timer_if_enabled, get_str,
get_transitioned_object_reader_with_tier_manager, inline_admission_shard_size, inline_erasure_shard_file_offset,
inline_erasure_shard_size, insert_str, is_deadlock_detection_enabled, is_err_object_not_found, is_err_version_not_found,
is_explicit_null_version, is_get_codec_streaming_base_enabled, is_get_small_object_direct_memory_enabled,
is_lock_optimization_enabled, issue3031_diag_enabled, join_all, known_put_object_storage_size, path_join_buf,
put_restore_opts, record_compression_total_memory, record_get_codec_streaming_gate_decision,
record_get_direct_memory_decision, record_get_object_pipeline_failure, record_get_object_pipeline_failure_for_path,
record_get_object_reader_path_observation, record_get_stage_duration_if_enabled, record_lock_acquire,
reduce_write_quorum_errs, release_materialized_read_lock, replication_write_may_pass_worm_gate,
republishes_data_movement_delete_marker, require_restore_operation_id, resolve_delete_version_state,
resolve_tiered_decommission_write_quorum_result, resolve_write_layout, restore_commit_operation_id_from_metadata,
restore_operation_id_from_metadata, send_event, set_disk_delete_creates_delete_marker,
@@ -2360,6 +2361,8 @@ fn is_restore_control_metadata(key: &str) -> bool {
.is_some_and(|remainder| remainder.is_empty())
|| rustfs_utils::http::internal_key_strip_suffix_prefix(key, SUFFIX_RESTORE_WORKER_LOCK)
.is_some_and(|remainder| remainder.is_empty())
|| rustfs_utils::http::internal_key_strip_suffix_prefix(key, SUFFIX_RESTORE_COPYBACK)
.is_some_and(|remainder| remainder.is_empty())
}
fn restore_metadata_update_preserves_protected_metadata(
@@ -4309,7 +4312,13 @@ impl SetDisks {
// state.
merge_replication_metadata_lww(&mut user_defined, &HashMap::new(), opts);
}
if expected_restore_operation_id.is_some() {
// Pending copy-back keeps its recovery identity until finalization.
if expected_restore_operation_id.is_some()
&& !user_defined
.get(metadata_keys::RESTORE)
.and_then(|value| rustfs_filemeta::parse_restore_obj_status(value).ok())
.is_some_and(|status| status.is_restore_in_progress == Some(true))
{
rustfs_utils::http::metadata_compat::remove_str(&mut user_defined, SUFFIX_RESTORE_OPERATION_ID);
rustfs_utils::http::metadata_compat::remove_str(&mut user_defined, SUFFIX_RESTORE_WORKER_LOCK);
}
@@ -4459,10 +4468,9 @@ impl SetDisks {
if let Some(disk) = disk_op
&& disk.is_online().await
{
match create_bitrot_writer_with_orig_volume(
match create_bitrot_writer(
is_inline_buffer,
Some(disk),
bucket,
RUSTFS_META_TMP_BUCKET,
&tmp_obj,
shard_file_size,
@@ -4694,19 +4702,29 @@ impl SetDisks {
inline_proof.to_owned(),
);
}
if opts.restore_copyback {
insert_str(
&mut user_defined,
SUFFIX_RESTORE_COPYBACK,
rustfs_utils::http::metadata_compat::RESTORE_COPYBACK_PROTOCOL_V1.to_string(),
);
}
fi.metadata = user_defined;
let put_tier_free_version_id =
if fi.version_id.is_none_or(|id| id.is_nil()) && !opts.data_movement && expected_restore_operation_id.is_none() {
// Current disks publish the same cleanup owner alongside a
// replaced null version. Keep the intent outside metadata:
// older disks persist unknown metadata keys, which splits
// the replacement's read-quorum identity during an upgrade.
let free_version_id = Uuid::new_v4();
fi.overwrite_tier_free_version_id = Some(free_version_id);
Some(free_version_id)
} else {
None
};
let put_tier_free_version_id = if fi.version_id.is_none_or(|id| id.is_nil())
&& !opts.data_movement
&& expected_restore_operation_id.is_none()
&& !opts.restore_copyback
{
// Current disks publish the same cleanup owner alongside a
// replaced null version. Keep the intent outside metadata:
// older disks persist unknown metadata keys, which splits
// the replacement's read-quorum identity during an upgrade.
let free_version_id = Uuid::new_v4();
fi.overwrite_tier_free_version_id = Some(free_version_id);
Some(free_version_id)
} else {
None
};
fi.mod_time = mod_time;
fi.size = w_size as i64;
fi.versioned = opts.versioned || opts.version_suspended;
@@ -6129,11 +6147,7 @@ impl SetDisks {
);
}
let all_local_disks_cleaned_tmp = all_commit_disks_local
&& online_disks.len() == commit_set.set_drive_count
&& online_disks.iter().all(Option::is_some)
&& !issue3031_diag_enabled();
if !tail_owns_tmp_cleanup && !all_local_disks_cleaned_tmp {
if !tail_owns_tmp_cleanup {
let cleanup_set = commit_set.clone();
let cleanup_tmp_dir = commit_tmp_dir.clone();
cleanup_runtime::spawn(commit_cleanup_reservation.clone(), async move {
@@ -11069,9 +11083,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
if let Some(err) = &gerr
&& goi.name.is_empty()
{
if (is_err_object_not_found(err) || is_err_version_not_found(err))
&& should_force_delete_marker_for_missing_version(&opts)
{
if should_force_delete_marker_for_missing_version(&opts) {
version_found = false;
} else {
if is_err_object_not_found(err) {
@@ -11598,7 +11610,14 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
if fi.transition_status == TRANSITION_COMPLETE {
return Ok(());
}
if fi
.metadata
.get(metadata_keys::RESTORE)
.and_then(|value| rustfs_filemeta::parse_restore_obj_status(value).ok())
.is_some_and(|status| status.is_restore_in_progress == Some(true))
{
return Ok(());
}
/*if fi.xlv1 {
if let Err(err) = self.heal_object(bucket, object, "", &HealOpts {no_lock: true, ..Default::default()}) {
return err.expect("err");
@@ -12102,7 +12121,9 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
let actual_fi = actual.fi();
let oi = ObjectInfo::from_file_info(actual_fi, bucket, object, opts.versioned || opts.version_suspended);
let mut restore_cleanup_info = oi.clone();
let restore_result: Result<()> = async {
crate::services::notification_sys::require_restore_copyback_fleet_capability().await?;
// The tier reader releases its own lease at EOF, before PUT or
// CompleteMultipartUpload necessarily reaches metadata quorum. Keep a
// second exact-generation lease for the whole restore so the final
@@ -12181,6 +12202,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
.put_object(bucket, object, &mut p_reader, &ropts)
.await
.map_err(|err| to_object_err(err, vec![bucket, object]))?;
restore_cleanup_info = restored_info.clone();
let restored_info = self_.finalize_restore_metadata(bucket, object, &restored_info, &opts).await?;
send_event(EventArgs {
event_name: EventName::ObjectRestoreCompleted.as_str().to_string(),
@@ -12300,6 +12322,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
bucket_lifecycle_lock_fence: opts.bucket_lifecycle_lock_fence.clone(),
user_defined: restore_commit_metadata,
no_lock: false,
restore_copyback: true,
decommission_capacity_admission: opts.decommission_capacity_admission.clone(),
// The composite ETag would otherwise be recomputed from the
// parts as they were written back, which for an encrypted or
@@ -12324,6 +12347,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
return Err(err);
}
};
restore_cleanup_info = restored_info.clone();
let restored_info = self_.finalize_restore_metadata(bucket, object, &restored_info, opts).await?;
send_event(EventArgs {
event_name: EventName::ObjectRestoreCompleted.as_str().to_string(),
@@ -12337,7 +12361,10 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
}
.await;
if let Err(primary_error) = restore_result {
if let Err(cleanup_error) = self_.update_restore_metadata(bucket, object, &oi, opts).await {
if let Err(cleanup_error) = self_
.update_restore_metadata(bucket, object, &restore_cleanup_info, opts)
.await
{
warn!(
event = EVENT_LIFECYCLE_RESTORE_CLEANUP,
component = LOG_COMPONENT_ECSTORE,
@@ -13310,8 +13337,8 @@ mod replication_quota_safety_tests {
let suspended_opts = ObjectOptions {
version_suspended: true,
delete_replication_config_snapshot: Some(Arc::new(DeleteReplicationConfigSnapshot::from_configs_for_test(
s3s::dto::VersioningConfiguration {
status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::SUSPENDED)),
rustfs_gateway_types::persistence::PersistedVersioningConfiguration {
status: Some("Suspended".to_owned()),
..Default::default()
},
None,
@@ -13944,11 +13971,11 @@ mod replication_lww_tests {
fn default_retention_snapshot(mode: &'static str) -> Arc<ObjectLockConfigSnapshot> {
Arc::new(ObjectLockConfigSnapshot::new(ObjectLockConfigState::Configured {
config: s3s::dto::ObjectLockConfiguration {
object_lock_enabled: Some(s3s::dto::ObjectLockEnabled::from_static(s3s::dto::ObjectLockEnabled::ENABLED)),
rule: Some(s3s::dto::ObjectLockRule {
default_retention: Some(s3s::dto::DefaultRetention {
mode: Some(s3s::dto::ObjectLockRetentionMode::from_static(mode)),
config: rustfs_gateway_types::persistence::PersistedObjectLockConfiguration {
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(rustfs_gateway_types::persistence::PersistedObjectLockRule {
default_retention: Some(rustfs_gateway_types::persistence::PersistedDefaultRetention {
mode: Some(mode.to_owned()),
days: Some(1),
years: None,
}),
@@ -13965,10 +13992,7 @@ mod replication_lww_tests {
/// timestamp the write passes and LWW judges it.
#[tokio::test]
async fn inbound_without_retention_timestamp_stays_rejected_under_bucket_default_retention() {
for mode in [
s3s::dto::ObjectLockRetentionMode::COMPLIANCE,
s3s::dto::ObjectLockRetentionMode::GOVERNANCE,
] {
for mode in ["COMPLIANCE", "GOVERNANCE"] {
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "lww-locked-default-retention";
let object = "object";
@@ -16235,7 +16259,7 @@ mod transition_commit_failure_tests {
use crate::services::tier::tier::TierConfigMgr;
use http::HeaderMap;
use rustfs_filemeta::{RestoreStatusOps as _, parse_restore_obj_status};
use s3s::dto::RestoreRequest;
use rustfs_lifecycle::RestoreRequest;
use tokio::io::AsyncReadExt;
pub(super) fn restore_operation_id_metadata(operation_id: Uuid) -> HashMap<String, String> {
@@ -18774,7 +18798,7 @@ mod transition_upload_integrity_tests {
for disk in disk_stores {
disk.make_volume(bucket).await.expect("bucket volume should be created");
}
let mut source = PutObjReader::from_vec(b"restore source body".repeat(1024));
let mut source = PutObjReader::from_vec(b"restore source body".repeat(64 * 1024));
set_disks
.put_object(bucket, object, &mut source, &ObjectOptions::default())
.await
@@ -18791,7 +18815,7 @@ mod transition_upload_integrity_tests {
.await
.expect("pending restore metadata should be installed");
let mut restored_reader = PutObjReader::from_vec(b"restored body".repeat(1024));
let mut restored_reader = PutObjReader::from_vec(b"restored body".repeat(64 * 1024));
set_disks
.put_object(
bucket,
@@ -18799,18 +18823,19 @@ mod transition_upload_integrity_tests {
&mut restored_reader,
&ObjectOptions {
user_defined: restore_metadata(operation_id, true),
restore_copyback: true,
..Default::default()
},
)
.await
.expect("matching restore commit should consume its operation id")
.expect("matching restore copy-back should retain its pending operation id")
}
fn restore_finalize_options(operation_id: Uuid) -> ObjectOptions {
ObjectOptions {
user_defined: restore_operation_id_metadata(operation_id),
transition: TransitionOptions {
restore_request: s3s::dto::RestoreRequest {
restore_request: rustfs_lifecycle::RestoreRequest {
days: Some(1),
..Default::default()
},
@@ -18820,17 +18845,87 @@ mod transition_upload_integrity_tests {
}
}
async fn assert_committed_restore_remains_pending(set_disks: &Arc<SetDisks>, bucket: &str, object: &str) {
#[tokio::test]
#[serial_test::serial]
async fn restore_cleanup_reclaims_committed_copyback_data_dir() {
let (temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "restore-cleanup-copyback-data-dir-bucket";
let object = "object.bin";
let operation_id = Uuid::new_v4();
let restored = write_committed_restore(&set_disks, &disk_stores, bucket, object, operation_id).await;
let data_dir = restored
.data_dir
.expect("committed copy-back must have a local data directory");
for temp_dir in &temp_dirs {
assert!(
temp_dir.path().join(bucket).join(object).join(data_dir.to_string()).exists(),
"committed copy-back must retain its local data directory before cleanup"
);
}
set_disks
.update_restore_metadata(
bucket,
object,
&restored,
&ObjectOptions {
user_defined: restore_operation_id_metadata(operation_id),
..Default::default()
},
)
.await
.expect("matching restore cleanup should remove the failed copy-back");
for temp_dir in &temp_dirs {
assert!(
!temp_dir.path().join(bucket).join(object).join(data_dir.to_string()).exists(),
"failed copy-back cleanup must reclaim its committed local data directory"
);
}
let cleaned = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("cleaned object metadata should remain readable");
assert!(!cleaned.user_defined.contains_key(rustfs_filemeta::metadata_keys::RESTORE));
assert!(
rustfs_utils::http::metadata_compat::get_consistent_str(
cleaned.user_defined.as_ref(),
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_COPYBACK,
)
.is_none(),
"failed copy-back cleanup must remove the local routing marker"
);
}
async fn assert_committed_restore_remains_pending(set_disks: &Arc<SetDisks>, bucket: &str, object: &str, operation_id: Uuid) {
let current = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("pending restore metadata should remain readable");
assert!(
restore_operation_id_from_metadata(current.user_defined.as_ref())
.expect("operation id metadata should parse")
.is_none(),
"successful restore commit must have consumed the operation id"
assert_eq!(
restore_operation_id_from_metadata(current.user_defined.as_ref()).expect("operation id metadata should parse"),
Some(operation_id),
"unfinished restore copy-back must retain its exact operation id"
);
let operation_id = operation_id.to_string();
for (suffix, expected) in [
(rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_OPERATION_ID, operation_id.as_str()),
(
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_WORKER_LOCK,
rustfs_utils::http::metadata_compat::RESTORE_WORKER_LOCK_PROTOCOL_V1,
),
] {
for prefix in [
rustfs_utils::http::RUSTFS_INTERNAL_PREFIX,
rustfs_utils::http::MINIO_INTERNAL_PREFIX,
] {
assert_eq!(
current.user_defined.get(&format!("{prefix}{suffix}")).map(String::as_str),
Some(expected),
"unfinished restore copy-back must retain each recovery marker alias"
);
}
}
assert!(
rustfs_filemeta::parse_restore_obj_status(
current
@@ -18888,7 +18983,7 @@ mod transition_upload_integrity_tests {
if error.kind() == std::io::ErrorKind::Other
&& error.to_string() == "restore finalization lock lost before metadata update"
));
assert_committed_restore_remains_pending(&set_disks, bucket, object).await;
assert_committed_restore_remains_pending(&set_disks, bucket, object, operation_id).await;
}
#[tokio::test]
@@ -18925,7 +19020,7 @@ mod transition_upload_integrity_tests {
..
}
));
assert_committed_restore_remains_pending(&set_disks, bucket, object).await;
assert_committed_restore_remains_pending(&set_disks, bucket, object, operation_id).await;
}
async fn assert_local_source_intact(set_disks: &Arc<SetDisks>, bucket: &str, object: &str, payload: &[u8]) {
@@ -19475,7 +19570,7 @@ mod transition_upload_integrity_tests {
no_lock: true,
part_number: Some(1),
transition: TransitionOptions {
restore_request: s3s::dto::RestoreRequest {
restore_request: rustfs_lifecycle::RestoreRequest {
days: Some(1),
..Default::default()
},
@@ -24148,29 +24243,20 @@ mod put_object_tmp_cleanup_tests {
(
"compliance",
HashMap::from([
(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(),
),
(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string()),
(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until.clone()),
]),
),
(
"governance",
HashMap::from([
(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
s3s::dto::ObjectLockRetentionMode::GOVERNANCE.to_string(),
),
(metadata_keys::OBJECT_LOCK_MODE.to_string(), "GOVERNANCE".to_string()),
(metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(), retain_until),
]),
),
(
"legal-hold",
HashMap::from([(
metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(),
s3s::dto::ObjectLockLegalHoldStatus::ON.to_string(),
)]),
HashMap::from([(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string())]),
),
];
@@ -24371,13 +24457,11 @@ mod put_object_tmp_cleanup_tests {
.expect("versioned PUT should return a version ID")
.to_string();
let default_compliance = ObjectLockConfigState::Configured {
config: s3s::dto::ObjectLockConfiguration {
object_lock_enabled: Some(s3s::dto::ObjectLockEnabled::from_static(s3s::dto::ObjectLockEnabled::ENABLED)),
rule: Some(s3s::dto::ObjectLockRule {
default_retention: Some(s3s::dto::DefaultRetention {
mode: Some(s3s::dto::ObjectLockRetentionMode::from_static(
s3s::dto::ObjectLockRetentionMode::COMPLIANCE,
)),
config: rustfs_gateway_types::persistence::PersistedObjectLockConfiguration {
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(rustfs_gateway_types::persistence::PersistedObjectLockRule {
default_retention: Some(rustfs_gateway_types::persistence::PersistedDefaultRetention {
mode: Some("COMPLIANCE".to_owned()),
days: Some(1),
years: None,
}),
@@ -24537,10 +24621,7 @@ mod put_object_tmp_cleanup_tests {
&ObjectOptions {
version_id: Some(destination_version.clone()),
versioned: true,
eval_metadata: Some(HashMap::from([(
metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(),
s3s::dto::ObjectLockLegalHoldStatus::ON.to_string(),
)])),
eval_metadata: Some(HashMap::from([(metadata_keys::OBJECT_LOCK_LEGAL_HOLD.to_string(), "ON".to_string())])),
..Default::default()
},
)
@@ -25448,7 +25529,7 @@ mod delete_objects_lock_gating_tests {
#[tokio::test]
async fn delete_objects_derives_per_object_versioning_from_the_request_snapshot() {
use s3s::dto::{BucketVersioningStatus, ExcludedPrefix, VersioningConfiguration};
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "batch-versioning-snapshot-bucket";
@@ -25460,10 +25541,8 @@ mod delete_objects_lock_gating_tests {
put_plain_object(&set_disks, bucket, "archive/unversioned-object").await;
let snapshot = Arc::new(DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
excluded_prefixes: Some(vec![ExcludedPrefix {
prefix: Some("archive/".to_string()),
}]),
status: Some("Enabled".to_owned()),
excluded_prefixes: Some(vec![Some("archive/".to_string())]),
..Default::default()
},
None,
@@ -25509,11 +25588,13 @@ mod delete_objects_lock_gating_tests {
#[tokio::test]
async fn batch_version_delete_uses_tags_read_under_the_delete_lock() {
use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
use s3s::dto::{
BucketVersioningStatus, DeleteReplication, DeleteReplicationStatus, Destination, ReplicationConfiguration,
ReplicationRule, ReplicationRuleFilter, ReplicationRuleStatus, Tag, VersioningConfiguration,
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use rustfs_gateway_types::persistence::{
PersistedReplicationConfiguration, PersistedReplicationDestination, PersistedReplicationFilter,
PersistedReplicationRule, PersistedReplicationStatus, PersistedReplicationTag,
};
use rustfs_utils::http::headers::AMZ_OBJECT_TAGGING;
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
let bucket = "batch-delete-locked-tags";
@@ -25549,23 +25630,23 @@ mod delete_objects_lock_gating_tests {
let snapshot = Arc::new(DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
rules: vec![PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
delete_replication: Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
}),
destination: Destination {
destination: PersistedReplicationDestination {
bucket: arn.to_string(),
..Default::default()
},
existing_object_replication: None,
filter: Some(ReplicationRuleFilter {
tag: Some(Tag {
filter: Some(PersistedReplicationFilter {
tag: Some(PersistedReplicationTag {
key: Some("generation".to_string()),
value: Some("locked".to_string()),
}),
@@ -25575,7 +25656,7 @@ mod delete_objects_lock_gating_tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}],
}),
));
@@ -25613,9 +25694,11 @@ mod delete_objects_lock_gating_tests {
#[tokio::test]
#[serial]
async fn lifecycle_delete_all_history_records_exact_replication_purge() {
use s3s::dto::{
BucketVersioningStatus, DeleteReplication, DeleteReplicationStatus, Destination, ReplicationConfiguration,
ReplicationRule, ReplicationRuleStatus, VersioningConfiguration,
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use rustfs_gateway_types::persistence::{
PersistedReplicationConfiguration, PersistedReplicationDestination, PersistedReplicationRule,
PersistedReplicationStatus,
};
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
@@ -25657,17 +25740,17 @@ mod delete_objects_lock_gating_tests {
let snapshot = Arc::new(DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
rules: vec![PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
delete_replication: Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
}),
destination: Destination {
destination: PersistedReplicationDestination {
bucket: arn.to_string(),
..Default::default()
},
@@ -25677,7 +25760,7 @@ mod delete_objects_lock_gating_tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}],
}),
));
@@ -25719,10 +25802,11 @@ mod delete_objects_lock_gating_tests {
#[tokio::test]
async fn synthetic_directory_delete_uses_decoded_prefix_and_marker_switch() {
use s3s::dto::{
BucketVersioningStatus, DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication,
DeleteReplicationStatus, Destination, ReplicationConfiguration, ReplicationRule, ReplicationRuleFilter,
ReplicationRuleStatus, VersioningConfiguration,
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
use rustfs_gateway_types::persistence::{
PersistedOptionalReplicationStatus, PersistedReplicationConfiguration, PersistedReplicationDestination,
PersistedReplicationFilter, PersistedReplicationRule, PersistedReplicationStatus,
};
let (_temp_dirs, disk_stores, set_disks) = hermetic_set_disks(4).await;
@@ -25736,24 +25820,24 @@ mod delete_objects_lock_gating_tests {
let snapshot = Arc::new(DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
status: Some("Enabled".to_owned()),
..Default::default()
},
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
delete_marker_replication: Some(DeleteMarkerReplication {
status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
rules: vec![PersistedReplicationRule {
delete_marker_replication: Some(PersistedOptionalReplicationStatus {
status: Some(Status::ENABLED.to_string()),
}),
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED),
delete_replication: Some(PersistedReplicationStatus {
status: Status::DISABLED.to_string(),
}),
destination: Destination {
destination: PersistedReplicationDestination {
bucket: arn.to_string(),
..Default::default()
},
existing_object_replication: None,
filter: Some(ReplicationRuleFilter {
filter: Some(PersistedReplicationFilter {
prefix: Some("photos/".to_string()),
..Default::default()
}),
@@ -25761,7 +25845,7 @@ mod delete_objects_lock_gating_tests {
prefix: None,
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}],
}),
));
@@ -25893,10 +25977,7 @@ mod delete_objects_lock_gating_tests {
// stat and the #4297 rejection.
let retain_until = OffsetDateTime::now_utc() + Duration::from_secs(60 * 60 * 24 * 30);
let mut user_defined = HashMap::new();
user_defined.insert(
metadata_keys::OBJECT_LOCK_MODE.to_string(),
s3s::dto::ObjectLockRetentionMode::COMPLIANCE.to_string(),
);
user_defined.insert(metadata_keys::OBJECT_LOCK_MODE.to_string(), "COMPLIANCE".to_string());
user_defined.insert(
metadata_keys::OBJECT_LOCK_RETAIN_UNTIL_DATE.to_string(),
retain_until
@@ -27526,9 +27607,10 @@ mod batch_delete_marker_tests {
#[tokio::test]
async fn batch_delete_explicit_marker_does_not_wait_for_pending_replication() {
use s3s::dto::{
BucketVersioningStatus, DeleteReplication, DeleteReplicationStatus, Destination, ReplicationConfiguration,
ReplicationRule, ReplicationRuleStatus, VersioningConfiguration,
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::{
PersistedReplicationConfiguration, PersistedReplicationDestination, PersistedReplicationRule,
PersistedReplicationStatus, PersistedVersioningConfiguration,
};
let (_dirs, store) = isolated_store_over_temp_disks().await;
@@ -27536,18 +27618,18 @@ mod batch_delete_marker_tests {
let (data_versions, markers) = marker_history(&store, bucket).await;
let arn = "arn:rustfs:replication:us-east-1:target:bucket";
let snapshot = Arc::new(DeleteReplicationConfigSnapshot::from_configs_for_test(
VersioningConfiguration {
status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
PersistedVersioningConfiguration {
status: Some("Enabled".to_owned()),
..Default::default()
},
Some(ReplicationConfiguration {
Some(PersistedReplicationConfiguration {
role: String::new(),
rules: vec![ReplicationRule {
rules: vec![PersistedReplicationRule {
delete_marker_replication: None,
delete_replication: Some(DeleteReplication {
status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
delete_replication: Some(PersistedReplicationStatus {
status: Status::ENABLED.to_string(),
}),
destination: Destination {
destination: PersistedReplicationDestination {
bucket: arn.to_string(),
..Default::default()
},
@@ -27557,7 +27639,7 @@ mod batch_delete_marker_tests {
prefix: Some(String::new()),
priority: Some(1),
source_selection_criteria: None,
status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
status: Status::ENABLED.to_string(),
}],
}),
));
+46 -3
View File
@@ -27,6 +27,7 @@ use std::sync::Arc;
struct RestoreFinalizeBarrierState {
bucket: String,
object: String,
before_lock: bool,
arrived: tokio::sync::Notify,
release: tokio::sync::Notify,
}
@@ -43,9 +44,18 @@ pub(in crate::set_disk) struct RestoreFinalizeBarrier {
#[cfg(all(test, feature = "test-util"))]
impl RestoreFinalizeBarrier {
pub(in crate::set_disk) fn install(bucket: &str, object: &str) -> Self {
Self::install_at(bucket, object, false)
}
pub(in crate::set_disk) fn install_before_lock(bucket: &str, object: &str) -> Self {
Self::install_at(bucket, object, true)
}
fn install_at(bucket: &str, object: &str, before_lock: bool) -> Self {
let state = Arc::new(RestoreFinalizeBarrierState {
bucket: bucket.to_string(),
object: object.to_string(),
before_lock,
arrived: tokio::sync::Notify::new(),
release: tokio::sync::Notify::new(),
});
@@ -81,13 +91,13 @@ impl Drop for RestoreFinalizeBarrier {
}
#[cfg(all(test, feature = "test-util"))]
async fn maybe_pause_restore_finalize(bucket: &str, object: &str) {
async fn maybe_pause_restore_finalize(bucket: &str, object: &str, before_lock: bool) {
let barrier = RESTORE_FINALIZE_BARRIER
.get_or_init(|| std::sync::Mutex::new(None))
.lock()
.expect("restore finalize barrier mutex should not poison")
.as_ref()
.filter(|barrier| barrier.bucket == bucket && barrier.object == object)
.filter(|barrier| barrier.bucket == bucket && barrier.object == object && barrier.before_lock == before_lock)
.cloned();
if let Some(barrier) = barrier {
barrier.arrived.notify_one();
@@ -150,6 +160,8 @@ impl SetDisks {
obj_info: &ObjectInfo,
opts: &ObjectOptions,
) -> Result<ObjectInfo> {
#[cfg(all(test, feature = "test-util"))]
maybe_pause_restore_finalize(bucket, object, true).await;
let expected = RestoreCleanupIdentity::from_object_info(obj_info);
let expected_operation_id = restore_operation_id_from_metadata(&opts.user_defined)?;
let expected_etag = obj_info
@@ -208,7 +220,7 @@ impl SetDisks {
return Err(Error::other("restored object changed before restore metadata finalization"));
}
#[cfg(all(test, feature = "test-util"))]
maybe_pause_restore_finalize(bucket, object).await;
maybe_pause_restore_finalize(bucket, object, false).await;
let restore_expiry =
lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), opts.transition.restore_request.days.unwrap_or(1));
fi.metadata.insert(
@@ -219,7 +231,14 @@ impl SetDisks {
}
.to_string(),
);
fi.transition_status = lifecycle::TRANSITION_COMPLETE.to_string();
rustfs_utils::http::metadata_compat::insert_str(
&mut fi.metadata,
rustfs_utils::http::metadata_compat::SUFFIX_TRANSITION_STATUS,
lifecycle::TRANSITION_COMPLETE.to_string(),
);
for suffix in [
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_COPYBACK,
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_OPERATION_ID,
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_WORKER_LOCK,
] {
@@ -315,10 +334,20 @@ impl SetDisks {
if !expected.matches_file_info(&fi, &expected_etag) {
return Ok(());
}
let copyback_data_dir = rustfs_utils::http::metadata_compat::get_consistent_str(
&fi.metadata,
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_COPYBACK,
)
.filter(|value| *value == rustfs_utils::http::metadata_compat::RESTORE_COPYBACK_PROTOCOL_V1)
.and_then(|_| fi.data_dir.filter(|data_dir| !data_dir.is_nil()));
ensure_restore_metadata_lock_held(bucket, object, opts, "restore_cleanup_metadata")?;
fi.metadata.remove(metadata_keys::RESTORE);
fi.metadata.remove(metadata_keys::RESTORE_EXPIRY_DAYS);
fi.metadata.remove(metadata_keys::RESTORE_REQUEST_DATE);
rustfs_utils::http::metadata_compat::remove_str(
&mut fi.metadata,
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_COPYBACK,
);
rustfs_utils::http::metadata_compat::remove_str(
&mut fi.metadata,
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_OPERATION_ID,
@@ -348,6 +377,20 @@ impl SetDisks {
)
.await?;
self.invalidate_get_object_metadata_cache(bucket, object).await;
if let Some(data_dir) = copyback_data_dir {
let cleanup = self
.commit_rename_data_dir_and_mark_capacity(
&disks,
bucket,
object,
&data_dir.to_string(),
"",
self.default_write_quorum(),
)
.await;
self.report_old_data_dir_cleanup(bucket, object, &data_dir.to_string(), &cleanup)
.await;
}
Ok(())
}
}
@@ -274,3 +274,452 @@ async fn prepared_snapshot_transition_duplicate_and_late_get_use_committed_remot
"late GET must route to the remote generation instead of reopening the released local source"
);
}
#[tokio::test]
#[serial_test::serial]
async fn single_part_restore_copyback_stays_pending_before_finalization() {
restore_copyback_stays_pending_before_finalization(false).await;
}
#[tokio::test]
#[serial_test::serial]
async fn multipart_restore_copyback_stays_pending_before_finalization() {
restore_copyback_stays_pending_before_finalization(true).await;
}
async fn restore_copyback_stays_pending_before_finalization(multipart: bool) {
use crate::bucket::lifecycle::bucket_lifecycle_ops::LifecycleOps as _;
use crate::set_disk::replication::RestoreFinalizeBarrier;
use crate::storage_api_contracts::multipart::{CompletePart, MultipartOperations as _};
use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
use rustfs_lifecycle::Lifecycle as _;
let (dirs, set_disks) = make_local_set_disks(4, 2).await;
let bucket = "restore-before-finalization-bucket";
let object = "object.bin";
let payload = vec![0x5au8; 6 * 1024 * 1024];
set_disks
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
let write_opts = ObjectOptions {
write_completion: WriteCompletion::TailDrained,
..Default::default()
};
let original = if multipart {
let upload = set_disks
.new_multipart_upload(bucket, object, &write_opts)
.await
.expect("source multipart upload should be created");
let mut parts = Vec::new();
for (index, bytes) in payload.chunks(5 * 1024 * 1024).enumerate() {
let mut reader = PutObjReader::from_vec(bytes.to_vec());
let part = set_disks
.put_object_part(bucket, object, &upload.upload_id, index + 1, &mut reader, &write_opts)
.await
.expect("source part should be written");
parts.push(CompletePart {
part_num: part.part_num,
etag: part.etag,
..Default::default()
});
}
set_disks
.clone()
.complete_multipart_upload(bucket, object, &upload.upload_id, parts, &write_opts)
.await
.expect("source multipart upload should complete")
} else {
let mut reader = PutObjReader::from_vec(payload.clone());
set_disks
.put_object(bucket, object, &mut reader, &write_opts)
.await
.expect("source object should be written")
};
let tier_name = format!("COLDTIER{}", &Uuid::new_v4().simple().to_string()[..8]).to_uppercase();
let backend = register_mock_tier(&set_disks.instance_ctx().tier_config_mgr(), &tier_name).await;
set_disks
.transition_object(
bucket,
object,
&ObjectOptions {
transition: TransitionOptions {
status: TRANSITION_PENDING.to_string(),
tier: tier_name.clone(),
etag: original.etag.clone().expect("source ETag should be present"),
..Default::default()
},
version_id: original.version_id.map(|version| version.to_string()),
mod_time: original.mod_time,
..Default::default()
},
)
.await
.expect("source should transition before restore");
let operation_id = Uuid::new_v4().to_string();
let mut operation_metadata = std::collections::HashMap::new();
rustfs_utils::http::metadata_compat::insert_str(
&mut operation_metadata,
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_OPERATION_ID,
operation_id.clone(),
);
rustfs_utils::http::metadata_compat::insert_str(
&mut operation_metadata,
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_WORKER_LOCK,
rustfs_utils::http::metadata_compat::RESTORE_WORKER_LOCK_PROTOCOL_V1.to_string(),
);
operation_metadata.insert(
rustfs_filemeta::metadata_keys::RESTORE.to_string(),
"ongoing-request=\"true\"".to_string(),
);
set_disks
.put_object_metadata(
bucket,
object,
&ObjectOptions {
eval_metadata: Some(operation_metadata.clone()),
..Default::default()
},
)
.await
.expect("the restore operation generation should be installed");
let barrier = RestoreFinalizeBarrier::install_before_lock(bucket, object);
let restore_set = Arc::clone(&set_disks);
let restore = tokio::spawn(async move {
let mut opts = ObjectOptions {
user_defined: operation_metadata,
..Default::default()
};
opts.transition.restore_request.days = Some(1);
restore_set.restore_transitioned_object(bucket, object, &opts).await
});
tokio::time::timeout(Duration::from_secs(30), barrier.wait_until_paused())
.await
.expect("copy-back must reach the gap before finalization acquires its lock");
let pending = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("HEAD must remain available before finalization");
let pending_data_dir = pending
.data_dir
.expect("pending copy-back must retain its local data directory");
for dir in &dirs {
assert!(
dir.path()
.join(bucket)
.join(object)
.join(pending_data_dir.to_string())
.exists(),
"pending copy-back must retain its committed local data directory"
);
}
let status = parse_restore_obj_status(
pending
.user_defined
.get(rustfs_filemeta::metadata_keys::RESTORE)
.expect("copy-back must retain restore metadata"),
)
.expect("pending restore status should parse");
assert_eq!(
pending.transitioned_object.status, TRANSITION_COMPLETE,
"copy-back must retain the remote identity needed by recovery and cleanup"
);
backend.set_unreachable(true).await;
let mut restored_before_finalization = Vec::new();
set_disks
.get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
.await
.expect("pending copy-back must be readable locally before metadata finalization")
.stream
.read_to_end(&mut restored_before_finalization)
.await
.expect("pending local body should drain");
assert_eq!(restored_before_finalization, payload, "pending local copy must preserve every byte");
set_disks
.transition_object(
bucket,
object,
&ObjectOptions {
transition: TransitionOptions {
status: TRANSITION_PENDING.to_string(),
tier: pending.transitioned_object.tier.clone(),
etag: pending.etag.clone().expect("pending ETag should be present"),
..Default::default()
},
version_id: pending.version_id.map(|version| version.to_string()),
mod_time: pending.mod_time,
..Default::default()
},
)
.await
.expect("a queued transition must skip the pending restore copy-back");
let after_queued_transition = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("restore metadata should remain readable after a queued transition");
assert_eq!(
after_queued_transition.transitioned_object.status, TRANSITION_COMPLETE,
"queued transition must not remove local copy-back data before finalization"
);
// Evaluate beyond any pre-copy deadline without sleeping or advancing the
// process clock. Pending copy-back must never become a restore-expiry candidate.
let lifecycle = PersistedLifecycleConfiguration::default();
let expiry_evaluation_time = OffsetDateTime::now_utc() + time::Duration::days(2);
let action = lifecycle
.eval_inner(&pending.to_lifecycle_opts(), expiry_evaluation_time, 0)
.await
.action;
barrier.release();
tokio::time::timeout(Duration::from_secs(30), restore)
.await
.expect("released restore should complete promptly")
.expect("restore task should join")
.expect("copy-back should finalize successfully");
assert!(status.on_going(), "copy-back must not advertise completion before metadata finalization");
assert!(status.expiry().is_none(), "pending copy-back must not expose a completed expiry");
assert!(
!action.delete_restored(),
"scanner must not reclaim copy-back before metadata finalization"
);
for (suffix, expected) in [
(rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_OPERATION_ID, operation_id.as_str()),
(
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_WORKER_LOCK,
rustfs_utils::http::metadata_compat::RESTORE_WORKER_LOCK_PROTOCOL_V1,
),
] {
for prefix in [
rustfs_utils::http::RUSTFS_INTERNAL_PREFIX,
rustfs_utils::http::MINIO_INTERNAL_PREFIX,
] {
assert_eq!(
pending.user_defined.get(&format!("{prefix}{suffix}")).map(String::as_str),
Some(expected),
"pending copy-back must retain each recovery marker alias"
);
}
}
let completed = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("finalized metadata should be readable");
assert_eq!(
completed.data_dir,
Some(pending_data_dir),
"finalization must retain the local copy-back data directory"
);
assert_eq!(
completed.transitioned_object.status, TRANSITION_COMPLETE,
"finalization must restore the completed transition state"
);
let completed_status = parse_restore_obj_status(
completed
.user_defined
.get(rustfs_filemeta::metadata_keys::RESTORE)
.expect("finalized restore status should be present"),
)
.expect("finalized restore status should parse");
assert!(!completed_status.on_going(), "finalization must publish completion");
assert!(completed_status.expiry().is_some(), "finalization must publish the completed expiry");
for suffix in [
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_OPERATION_ID,
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_WORKER_LOCK,
] {
for prefix in [
rustfs_utils::http::RUSTFS_INTERNAL_PREFIX,
rustfs_utils::http::MINIO_INTERNAL_PREFIX,
] {
assert!(
!completed.user_defined.contains_key(&format!("{prefix}{suffix}")),
"finalization must remove every recovery marker alias"
);
}
}
let completed_action = lifecycle
.eval_inner(&completed.to_lifecycle_opts(), expiry_evaluation_time, 0)
.await
.action;
assert!(
completed_action.delete_restored(),
"the same lifecycle evaluator must expire a completed restore beyond its deadline"
);
backend.set_unreachable(true).await;
let mut restored = Vec::new();
set_disks
.get_object_reader(bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
.await
.expect("completed restore must be readable with the remote tier unavailable")
.stream
.read_to_end(&mut restored)
.await
.expect("local restored body should drain");
assert_eq!(restored, payload, "finalized local copy must preserve every byte");
for dir in &dirs {
assert!(
dir.path()
.join(bucket)
.join(object)
.join(pending_data_dir.to_string())
.exists(),
"completed restore must retain its local data directory"
);
}
}
#[tokio::test]
#[serial_test::serial]
async fn restore_copyback_overwrite_preserves_remote_cleanup_owner() {
assert_transition_overwrite_preserves_remote_cleanup_owner(true).await;
}
#[tokio::test]
#[serial_test::serial]
async fn completed_transition_overwrite_preserves_remote_cleanup_owner() {
assert_transition_overwrite_preserves_remote_cleanup_owner(false).await;
}
async fn assert_transition_overwrite_preserves_remote_cleanup_owner(during_copyback: bool) {
use crate::set_disk::replication::RestoreFinalizeBarrier;
let (dirs, set_disks) = make_local_set_disks(4, 2).await;
let bucket = "restore-copyback-cleanup-owner";
let object = "object.bin";
let payload = vec![0x37u8; 6 * 1024 * 1024];
let write_opts = ObjectOptions {
write_completion: WriteCompletion::TailDrained,
..Default::default()
};
set_disks
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("cleanup probe bucket should be created");
let mut reader = PutObjReader::from_vec(payload);
let original = set_disks
.put_object(bucket, object, &mut reader, &write_opts)
.await
.expect("cleanup probe source should be written");
let tier_name = format!("COLDTIER{}", &Uuid::new_v4().simple().to_string()[..8]).to_uppercase();
let backend = register_mock_tier(&set_disks.instance_ctx().tier_config_mgr(), &tier_name).await;
set_disks
.transition_object(
bucket,
object,
&ObjectOptions {
transition: TransitionOptions {
status: TRANSITION_PENDING.to_string(),
tier: tier_name.clone(),
etag: original.etag.clone().expect("source ETag should be present"),
..Default::default()
},
mod_time: original.mod_time,
..Default::default()
},
)
.await
.expect("cleanup probe source should transition");
let source = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("transitioned cleanup source should resolve");
assert_eq!(backend.object_count().await, 1, "transition must retain the real remote cleanup source");
let mut barrier = None;
let mut restore = None;
if during_copyback {
let operation_id = Uuid::new_v4();
let mut metadata = std::collections::HashMap::new();
rustfs_utils::http::metadata_compat::insert_str(
&mut metadata,
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_OPERATION_ID,
operation_id.to_string(),
);
rustfs_utils::http::metadata_compat::insert_str(
&mut metadata,
rustfs_utils::http::metadata_compat::SUFFIX_RESTORE_WORKER_LOCK,
rustfs_utils::http::metadata_compat::RESTORE_WORKER_LOCK_PROTOCOL_V1.to_string(),
);
metadata.insert(
rustfs_filemeta::metadata_keys::RESTORE.to_string(),
"ongoing-request=\"true\"".to_string(),
);
set_disks
.put_object_metadata(
bucket,
object,
&ObjectOptions {
eval_metadata: Some(metadata.clone()),
..Default::default()
},
)
.await
.expect("cleanup probe restore generation should be installed");
let installed = RestoreFinalizeBarrier::install_before_lock(bucket, object);
let restore_set = Arc::clone(&set_disks);
restore = Some(tokio::spawn(async move {
let mut opts = ObjectOptions {
user_defined: metadata,
..Default::default()
};
opts.transition.restore_request.days = Some(1);
restore_set.restore_transitioned_object(bucket, object, &opts).await
}));
tokio::time::timeout(Duration::from_secs(30), installed.wait_until_paused())
.await
.expect("real restore should publish local bytes before finalization");
let pending = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("copy-back metadata should resolve");
assert!(pending.restore_ongoing, "probe must hit an unfinished real restore");
barrier = Some(installed);
}
let mut replacement = PutObjReader::from_vec(vec![0x69u8; 6 * 1024 * 1024]);
set_disks
.put_object(bucket, object, &mut replacement, &write_opts)
.await
.expect("ordinary overwrite must succeed while restore finalization is paused");
if let Some(barrier) = barrier {
barrier.release();
}
if let Some(restore) = restore {
tokio::time::timeout(Duration::from_secs(30), restore)
.await
.expect("superseded restore must finish")
.expect("superseded restore must not panic")
.expect_err("restore finalization must not revive the overwritten generation");
}
let mut cleanup_owners = Vec::new();
for dir in &dirs {
let encoded = match tokio::fs::read(dir.path().join(bucket).join(object).join(crate::disk::STORAGE_FORMAT_FILE)).await {
Ok(encoded) => encoded,
Err(err) if err.kind() == std::io::ErrorKind::NotFound => {
cleanup_owners.push(0);
continue;
}
Err(err) => panic!("cleanup source metadata should be readable: {err}"),
};
let versions = rustfs_filemeta::FileMeta::load(&encoded)
.expect("cleanup source metadata should decode")
.get_file_info_versions(bucket, object, false)
.expect("cleanup source versions should decode");
let count = versions
.free_versions
.iter()
.filter(|version| {
version.transition_tier == source.transitioned_object.tier
&& version.transitioned_objname == source.transitioned_object.name
})
.count();
cleanup_owners.push(count);
}
let remote_objects = backend.object_count().await;
println!("restore_cleanup_probe during_copyback={during_copyback} owners={cleanup_owners:?} remote_objects={remote_objects}");
assert!(
remote_objects == 0 || cleanup_owners.iter().filter(|count| **count > 0).count() >= 3,
"ordinary overwrite must remove the remote object or retain its cleanup owner at write quorum: during_copyback={during_copyback}, owners={cleanup_owners:?}, remote_objects={remote_objects}"
);
}
@@ -44,16 +44,15 @@ pub(crate) mod lifecycle {
pub(crate) mod list {
pub(crate) use rustfs_storage_api::{
AuthoritativeListOperations, ListObjectVersionsInfo as StorageListObjectVersionsInfo, ListObjectsInfo,
ListObjectsV2Info as StorageListObjectsV2Info, ListOperations, ObjectInfoOrErr as StorageObjectInfoOrErr, VersionMarker,
WalkOptions as StorageWalkOptions, WalkVersionsSortOrder,
ListObjectVersionsInfo as StorageListObjectVersionsInfo, ListObjectsInfo, ListObjectsV2Info as StorageListObjectsV2Info,
ListOperations, ObjectInfoOrErr as StorageObjectInfoOrErr, VersionMarker, WalkOptions as StorageWalkOptions,
WalkVersionsSortOrder,
};
}
pub(crate) mod multipart {
pub(crate) use rustfs_storage_api::{
CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartInfo, MultipartOperations, MultipartUploadInspection,
MultipartUploadResult, PartInfo,
CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartInfo, MultipartOperations, MultipartUploadResult, PartInfo,
};
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More