Keep native publication recording in main's native_io helpers and resource
writers, and adapt merged tests to the owned Body, Bytes write input, and
fallible find_disk.
* feat(scanner): share node admission with background heal
* fix(scanner): drain candidates after listing completion
* fix(scanner): yield cache leases for shared node admission
* test(scanner): attest adaptive admission in process receipts
* fix(concurrency): declare Tokio time dependency
* feat(concurrency): retain bounded physical resource subsets
* fix(scanner): retain resource ownership in native filesystem jobs
* feat(concurrency): acquire physical groups before node work
* feat(scanner): gate background reads by physical device
* feat(concurrency): combine preallocated resource selections
* feat(scanner): share physical topology across disk plans
* fix(storage): retain only native disk resource subsets
* feat(storage): admit auxiliary physical resources without waiting
* feat(heal): admit physical resources before object repair
* fix(storage): retain resources during native format probes
* fix(storage): retain resources through direct writer I/O
* fix(storage): retain resources through buffered writer I/O
* fix(storage): retain physical resources through streamed reads
* fix(storage): retain resources through native filesystem jobs
* fix(storage): retain physical ownership during bitrot reads
* fix(ecstore): retain resources during positioned reads
* fix(ecstore): retain staged metadata write ownership
* fix(ecstore): retain native io_uring read ownership
* fix(ecstore): enforce receiver-side background admission
* fix(ecstore): retain resources through whole-file native I/O
* fix(ecstore): retain physical ownership during durable rename sync
* fix(storage): account for retained background stream buffers
* fix(storage): retain memory charges for background read results
* fix(storage): account for native io_uring read allocations
* fix(storage): retain memory charges for direct-write buffers
* fix(ecstore): account for heal and RPC reader buffers
* test(scanner): verify samples and clean concurrent collectors
* fix(storage): retain inline and HTTP writer memory reservations
* perf(rio): reuse reserved HTTP writer storage
* fix(rpc): reserve background scanner pipe storage
* perf(rpc): reuse reserved response reader storage
* feat(scanner): measure cache copy allocation requests
* fix(rpc): budget namespace scanner responses
* fix(scanner): retain the enabled supervisor task handle
* fix(scanner): budget namespace frame read staging
* fix(scanner): account namespace envelope decode storage
* fix(heal): retain and join the MRF consumer on shutdown
* test(heal): isolate process-global MRF consumer fixture
* fix(heal): join manager workers during shutdown
* test(heal): verify shutdown at native read boundaries
* fix(heal): drain channel commands before stopping MRF
* fix(heal): admit resume scans and retain replacement workers
* build: pin profiling sources and trust fixed ownership dependencies
* fix(heal): yield node admission while accepted native work drains
* fix(startup): retain shutdown work and preserve deadline failures
* fix(heal): admit inventory probes and retain replacement work
* fix(shutdown): preserve maintenance and frontend join failures
* fix(shutdown): retain HTTP child tasks through real retirement
* fix(shutdown): join actual TLS reload work before retirement
* test(multipart): preserve failures before fault barriers
* fix: keep physical admission inside repository guards
Route the multipart lock diagnostics filter through the e2e storage
compatibility module, reuse the parent module's s3s body aliases in the
namespace memory test, and share one Reed-Solomon verify error mapping
across the reserved-buffer and default paths.
* fix: reconcile physical admission with main's new call sites
Main added a source-admission metadata write layer, a CAS write fence,
fixture writers and scanner handoff tests against the old borrowed-slice
and node-permit shapes. Pass owned Bytes through the new metadata layer,
answer the scanner handoff with ScannerNodePermit, gate the test-only
disk lookup re-export, and clear the lint findings that stable 1.99
reports for the shutdown and admission code without suppressions.
* test(concurrency): name reversed attachment groups clearly
* fix(ecstore): box disk operation futures below admission layers
Every LocalDiskWrapper operation reaches the LocalDisk future through the
health, admission and timeout layers (track_cleanup_mutation, the
track_disk_health family, the io-permit and native-resource admission
split, and track_admitted_disk_operation). Each layer embedded that future,
and unoptimized builds copy it into every awaiting frame, so one local
delete cost about 1.3 MiB of stack and 16 tests overflowed libtest's
default 2 MiB stack.
Box the operation future once where track_admitted_disk_operation creates
it, through a non-async helper so the temporary never stays in a polled
frame. The layers above now hold only a pointer; the cost is one
allocation per local disk operation.
* fix(heal): box object heal futures below admission
The admission wrappers took the object heal future by value, so unoptimized
builds copied it into three awaiting frames, about 650 KiB per heal. Create
it through a non-async helper instead, leaving only a pointer in those frames.
* test(ecstore): count the I/O backend's root FD owner
Sync main 84e24e767 into gateway/integration. Ports main-side additions to
the integration boundaries: the tier deletion diagnostics admin handlers move
to the owned admin request/response types, and the new lifecycle rule checks
in ecstore use the persisted lifecycle types instead of s3s DTOs. Keeps the
gateway git source in deny.toml while dropping the smithy-rs pin main retired,
and refreshes the test-count baseline for the merged tree.
* chore(deps): refresh crates and adopt released aws-sigv4
Run two update/upgrade rounds for the main and standalone fuzz workspaces.
Replace the temporary SigV4a Git override with aws-sigv4 1.6.4 from the
registry and remove its obsolete source permission.
Pin OpenID Connect to e20ddbb01ca68356f2979bc125f5613ab45b4e7e. Upgrade
async-nats to 0.51.0, hotpath to 0.30.0, libunftp to 0.24.0, and the TOML
test dependency to 1.1.8. Retain Arrow/Parquet 59.3.2 for DataFusion 55.2.0
and keep the fuzz PKCS#1 version compatible with pinned RSA rc.18.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): retain safe openidconnect pin against HMAC panic
The e20ddbb pin verifies HMAC signatures with mac.verify(signature.into()),
which panics in generic-array on malformed length instead of returning
InvalidIdentityToken. Retain the previous reviewed pin that uses
verify_slice and add a malformed-signature regression covering HS256,
HS384, and HS512.
* chore(deps): upgrade sysinfo to 0.40.0
Update the main and standalone fuzz lockfiles. Adapt fallible system
queries, CPU usage accessors, and OS-native network interface names.
Preserve process sampling fallbacks and report unavailable diagnostic
sources without panicking. Cover missing host and process collectors.
Validation: 167 focused tests passed; production and all five fuzz targets
passed cargo check. Formatting, logging guardrails, and the existing
dependency policy passed. Linux and Windows runtime checks were not run.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ci): use the admin error facade for sysinfo failures
Keep the sysinfo initialization error behind the existing admin S3 facade.
The facade builds the same error code and message without expanding the
direct S3 error-macro surface guarded by the migration ratchet.
Validation: S3 footprint guard, formatting, whitespace, and production
cargo check with sftp, hotpath, and connect_top_net passed.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh hotpath and ctutils (#9055)
Run two cargo update/upgrade rounds in both the main and fuzz workspaces
on top of PR #9047. Upgrade the hotpath family to 0.30.1 and ctutils to
0.4.7. Retain fuzz pkcs1 0.8.0-rc.4 for the pinned RSA 0.10.0-rc.18;
rc.5 fails to compile against that RSA version.
Validation: 1096 library tests passed with 19 skipped; upstream ctutils
and hotpath-drain property/concurrency suites passed. Production and all
five fuzz targets passed cargo check. Formatting, S3 footprint, and the
existing dependency policy passed.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Source: main 4e48c390e7
Target: gateway/integration 1000860cb8
Conflicts resolved, keeping both sides:
- crates/ecstore/src/disk/{fs,local}.rs: main's batched rollback writes,
unified blocking write path and relocated open_file, with integration's
native publication records restored at each publication point and the
publication_history field kept beside main's multipart proof authority.
- crates/ecstore/src/set_disk/ops/multipart.rs: union of the imports.
- rustfs/src/storage/rpc/node_service.rs: main's MRF lifecycle reader
capability branch; the heal capability keeps integration's restore
copy-back probe.
- rustfs/src/admin/handlers/{audit,event,plugins_instances}.rs: main's
target_read inventory snapshot on integration's owned request boundary;
the new event.rs inventory test now builds owned requests.
- scripts/check_s3s_footprint.sh: integration's baseline-file version.
- .config/e2e-{full,smoke}-selection.txt: digests recomputed over the
union of both heads' CI listings (full 700 darwin / 708 linux, smoke 232).
Also migrated main's new rebalance delete-all expiry test to the persisted
lifecycle types integration uses in ecstore, and raised the test-count
baseline to the merged counts (every decrease comes from main).
* chore(deps): upgrade russh to 0.65.0 and refresh crates
Run two cargo update and cargo upgrade rounds from origin/main.
Adapt SFTP public-key callbacks to PublicKeyOffer and VerifiedPublicKey,
preserving explicit rejection and password-only authentication.
Cover the rejection policy with a pinned-host-key SSH handshake test.
Refresh Arrow, Parquet, Brotli, ZIP, and compatible dependencies.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): route legacy zstd consumers to fixed codecs
Replace the old registry implementation with a private version-constraint
adapter backed only by zstd 0.14.3 and zstd-safe 8.2.0. Preserve legacy
features and Avro, Arrow IPC, Parquet, and Kafka formats.
Cover abandoned contexts, legacy decode errors, seekable bounds, stored
frames, consumer interoperability, and resolved implementation versions.
Record the adapter removal condition in the compatibility register.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Replace the yanked crypto-bigint 0.7.7 release and raise ctutils to its required 0.4.6 minimum. Preserve all other package resolutions, features and dependency policy.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): read publication logs incrementally
Retain the complete matching lock history while reading only appended bytes after an initial scan before remote DELETE is held. Keep the original field projection, file ordering, polling cadence, lock deadlines and same-attempt assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): remove redundant publication fixture clone
Serialize the existing root by reference through json! without cloning the Value. Keep all compatibility rows and assertions unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Source: main 0f24511535
Target: gateway/integration 8cf6187ff4
Resolve the e2e-full selection digest conflict by recomputing both
platform digests over the merged nextest listing (685 darwin / 693
linux): main adds two rpc_fault_proxy unit tests, integration keeps its
dual-stack differential and anonymous-access additions. Raise the rustfs
test-count baseline row to the merged count (5932).
Run two cargo update and cargo upgrade rounds from origin/main.
Update mimalloc to 0.7.2, hotpath to 0.29.1, and ZIP to 9.1.0,
with compatible crypto, compression, and smallvec updates.
Retain the generic-array version required by crypto-common 0.1.7.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rio): isolate internode streams with pooled HTTP/1.1
Default the internode HTTP data plane to reusable HTTP/1.1 connections so paused shard bodies cannot consume shared HTTP/2 receive credit. Retain explicit auto negotiation and the existing TLS, pooling, and recovery paths.
Refs rustfs/rustfs#8843 and rustfs/backlog#2934.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(rpc): frame bounded shard reads for connection reuse
Preserve the bounded read cap and stream-to-EOF sentinel while making connection completion independent of a chunked terminator. Cover exact reads without EOF polling, pooled TLS reuse, and partial-body truncation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): migrate signer and OIDC to patched crypto
Pin the requested signer fix and upstream OIDC/OAuth2 crypto migrations to immutable commits. Preserve registry SDK type identity, remove the old bigint and elliptic-curve chains, and verify independent login state, nonce, and PKCE values.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): use the resumed decommission completion budget
* test(heal): exercise checkpoints written by rc.5
* test(heal): route upgrade fixtures through the storage facade
* fix(deps): update ctutils to address constant-time advisory
* test(lifecycle): gate recovery I/O while advancing test time
Brings in #8825, #8831, #8836 and #8839. Conflicts were import lists and test blocks (both sides kept), plus the new restore_copyback request flag next to main's data-movement field. Main replaced completed_multipart_object_part with prepare_complete_multipart_parts, so the stale import is dropped.
Main's xl.meta fast-path scanner (inspect_v2_object_slice) predates #8836 and classified a transitioned version with a pending v1 restore copyback as having no physical data dir, which made find_unshared_data_dir_for_version drop the copyback data dir. The scanner now reads the copyback marker with get_consistent_bytes semantics (both aliases, ASCII case-insensitive, empty or conflicting values count as absent), and uses_data_dir keeps main's allocation-free transition check plus #8836's copyback exclusion. find_unshared_data_dir_for_version_keeps_pending_copyback_data_dir failed on the plain merge and passes now.
Run two cargo update and cargo upgrade rounds from origin/main.
Upgrade the hotpath crate family to 0.29.0 and refresh compatible
AWS, Redis, profiling, crypto, UUID, and certificate dependencies.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Picks up the 20 main commits since 62b6f5f3f. The one conflict is the tier free-version recovery test setup: it keeps the integration-only fixture isolation (background recovery disabled while the fixture starts) and main's mock tier registration (#8804 skips scans with no tier). The e2e selections are unchanged by these commits; the test-count baseline is regenerated.
* chore(deps): upgrade workspace parking_lot to 0.13
Move all six direct workspace consumers to parking_lot 0.13 while
retaining the 0.12 dependency line required by upstream crates.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(connect): stabilize client failure classification
Use an ephemeral loopback listener that explicitly rejects proxy CONNECT
and direct TLS connection establishment instead of assuming fixed ports
will promptly refuse connections. Give classification its own bounded
transport and measurement budget while retaining the timeout tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh GCS, profiling and allocator dependencies
Update published GCS, Pyroscope, mimalloc, and PPMd crate versions.
Align the pinned erasure-fixture test recursion limit with ECStore
to compile GCS-enabled test targets.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): update AWS SDKs, Tokio and memmap2
Refresh compatible AWS service and Smithy dependencies, Tokio timers,
and memory mapping support with the regenerated lockfile.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The scheduled sync has stopped on conflicts since 2026-10-19. Resolved by keeping both sides: main's restructured delete-rollback path in disk/local.rs with the integration-only native-publication record, main's per-instance region with the rustfs-s3-types Region, main's new lifecycle, rebalance and admin code ported to the persisted lifecycle types and rustfs-s3-types errors, both sides' new scanner tests, and the union of the access_consumer imports. The e2e smoke, full (darwin) and repl-nightly selections and the test-count baseline are regenerated from the merged tree.
* refactor(s3select): run queries on an engine request built from gateway DTO shapes
Part of T1.5 (rustfs/backlog#2746), batch 2.
* fix(s3select): keep the select request conversion in the app layer
The layer guard forbids app -> storage; the conversion moves from storage/s3_compat/select.rs to app/select_request.rs unchanged.
* refactor(ecstore): move bucket configs onto gateway persisted types
Finish the ECStore batch of T1.4 (rustfs/backlog#2745). The remaining
eleven bucket configuration families (Website, CORS, Tagging, Versioning,
ObjectLock, Encryption, Notification, Logging, Accelerate, RequestPayment,
PublicAccessBlock) now read and write their stored XML through the
rustfs-gateway-types persistence and cors_tagging codecs instead of the
s3s codec, so the legacy s3s bucket-config codec and its
s3gate-metadata-xml compat marker are removed.
- ecstore: BucketConfigXml is implemented by the gateway codec for every
family; metadata, metadata_sys, versioning, object lock, replication and
lifecycle consumers take the persisted types. Region moves to
rustfs_s3_types::Region, and the ETag conversion to s3s moves to the
rustfs edge with its tests. Only warm_backend.rs and warm_backend_s3.rs
still reference s3s, held for T1.5 (rustfs/backlog#2746).
- scanner: object lock cache, item actions and data usage use the
persisted types.
- rustfs: the legacy S3 edge converts each family between s3s DTOs and the
persisted types in storage/storage_api.rs by writing one codec's XML and
reading it with the other; Tagging converts field by field.
- Bump the rustfs-gateway-types pin to 21283e87.
The 39 persisted-byte goldens are unchanged and pass, so a document the
old codec wrote reads back and re-encodes to the same bytes. s3s footprint:
files 181 -> 146, ecstore_files 28 -> 2, s3_error_lines unchanged.
* test(app): prepare the PutBucketEncryption config from the request
The uninitialized-store test passed an empty persisted configuration, which the use case now validates before it reaches the store check, so it answered MalformedXML instead of InternalError. Convert the request's configuration the way the S3 edge does.
* style(app): format the PutBucketEncryption test
* test(heal): add failing regressions for a source-independent absent verdict
Add the regression tests for backlog#2900 that fail on the current code,
where the same object-level not-found is treated as deleted on the plain
heal path but as recreate_failed on the recreate path, and where the
erasure-set healer, the task predicates and the error helpers disagree on
wrapped not-found and transient inputs.
Also add the per-object disposition cache module with its unit tests, the
manager admission tests for it, the compat register entry for the display
boundary, and the admission-invariant notes. The module and the manager
tests are not declared yet, so nothing is wired into the build.
* fix(heal): decide object absence from storage evidence, not request source
Replace the five heal error classifiers (erasure-set healer outcome
mapping, task-level not-found and lock predicates, error.rs recovery
helpers and the storage existence-check matcher) with one typed
classify_heal_error. An object-level not-found from the locked storage
read is Absent for every source and every recreate_missing value; the
recreate path, the usage-observation and scanner object-dir whitelists
are gone, and display-text matching survives only in the marked
classify_error_display boundary.
Add a bounded, expiring per-object disposition memory consulted before
queue admission so a best-effort producer cannot requeue an absent or
terminally failed object on every tick; durable MRF, admin, forced,
high-priority and dry-run requests bypass it. The read-repair producer no
longer forces recreate_missing.
* fix(heal): keep rendered grace windows out of task retries
A grace window rendered into a task message belongs to an attempt that
already finished; its owner schedules the next one. Only typed grace
errors replay the task, as before the classifier merge. Update the
ecstore read-repair black-box test: the producer no longer forces
recreate_missing.
* fix(heal): classify ambiguous legacy delete residue as permanent explicitly
* fix(heal): scope disposition memory and bound absence parsing
Keep lifecycle rules, restore requests, Object Lock, and scanner cached tags in domain types. Convert at the existing HTTP boundary while preserving legacy XML, scanner cache bytes, date ordering, and tombstones.
Linux test validation before the final import-path-only adjustment:
cargo nextest run -p rustfs-ecstore bucket_config_goldens
39 passed; 6211 skipped by selector.
cargo nextest run --lib -p rustfs -E 'test(/test_zero_rule_lifecycle_tombstone_recognition/) | test(/test_local_lifecycle_staleness_axis_selection/) | test(/test_lifecycle_staleness_preserves_legacy_submillisecond_axis/) | test(/test_merge_incoming_lifecycle/) | test(/test_lifecycle_expiry_statement_matrix/) | test(/test_lifecycle_expiry_subset_xml/) | test(/put_bucket_lifecycle_validation_errors_keep_their_s3_code/) | test(/execute_put_bucket_lifecycle_configuration_rejects_missing_configuration/) | test(/execute_restore_object_rejects_missing_restore_request/) | test(/execute_restore_object_rejects_select_type/) | test(/execute_restore_object_maps_failures_to_typed_s3_errors/) | test(/execute_restore_object_returns_internal_error_when_store_uninitialized/) | test(/restore_request_conversion_preserves_legacy_output_values/) | test(/restore_request_conversion_preserves_presence_and_raw_enums/) | test(/restore_request_conversion_distinguishes_absent_and_empty_containers/)'
25 passed; 5318 skipped by selector.
cargo nextest run --profile ci -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-replication -p rustfs-scanner
7872 passed; 21 skipped under the existing CI profile.
Previously verified mutation controls: 39 production faults, 41 RED/GREEN pairs.
cargo clippy --locked -p rustfs -p rustfs-ecstore -p rustfs-lifecycle -p rustfs-scanner --all-targets -- -D warnings
Passed in 40m 45s before the import-path adjustment, then in 1m 59s on
the final source with warnings denied. The final adjustment imports the
existing helper through the object domain facade; fmt and the architecture
check also passed.
Part of rustfs/backlog#2745. The remaining ecstore configuration families follow separately.
* chore(deps): refresh workspace dependencies
Run cargo update and cargo upgrade twice from origin/main.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): pin Pulsar 6.9.0 until recovery is bounded
Keep Pulsar on 6.9.0 after reproducing background recovery and lookup activity after delivery timeout and target destruction in 6.9.1. Use an exact requirement to prevent a later cargo update from selecting the affected release.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Keep the migrated compatibility adapters in their active modules while bringing in upstream fixes. Preserve native publication history on the new create, rename and borrowed-open paths.
Validation:
- `cargo nextest run --locked -p rustfs-ecstore --lib -E 'test(disk::os::native_creation_history_tests::) | test(disk::local::test::borrowed_open_cancellation_retains_started_history_before_execution) | test(disk::local::test::owned_direct_child_open_attempt_consumes_native_history)' --no-fail-fast`: 9/9 passed after restoring all 12 mutation cases at cea78f2590645eb0d303cd7cf56f4dd0d9c00e87; these source blobs are unchanged in this merge.
- `cargo nextest run --locked -p rustfs-s3select-query --lib -E 'test(dispatcher::json_document_schema::tests::)' --no-fail-fast`: 5/5 passed at 5421ba682d1329cc7ebeaac7c6a863da2cea1996, with the same tree.
- `cargo fmt --all --check`: passed.
- `./scripts/check_test_count_ratchet.sh --update`: 52 crates, 22007 tests, 165 ignored.
- Profile baselines use seven actual Darwin listings and seven actual Linux listings.
- Earlier conflict snapshot f5e0cdf0c58aed1ebe53b9243cf52f94832ab049: `cargo nextest run --locked -p rustfs-ecstore -p rustfs --lib -E 'test(disk::os::) | test(bucket::metadata_sys::) | test(server::legacy_compat::) | test(server::layer::tests::request_logging_watchdog) | test(server::http::tests::conditional_object_http::) | test(storage::helper::tests::operation_helper)' --no-fail-fast` passed 218/218 before the borrowed-open fix. This is historical evidence.
* wip: T1.4 replication family
* refactor(replication): move the replication config onto gateway types
The replication configuration family leaves s3s: rustfs-replication no longer depends on s3s, and ECStore bucket metadata, the scanner and the rustfs app/admin layers hold rustfs_gateway_types::persistence::PersistedReplicationConfiguration. The persisted XML goes through the gateway persistence codec behind a new BucketConfigXml trait (the 39 byte goldens pass with unchanged fixtures); the legacy s3s edge converts once in storage/ecfs.rs through storage/s3_compat/replication.rs; the scanner data usage cache keeps its MessagePack bytes through a serde mirror pinned by bytes recorded from s3s 5761ddfe. rustfs-gateway-types 0.32.0 is pinned once (rev 772d3385). Refs rustfs/backlog#2745.
* test: use S3 serializer for list response tests
* feat(diagnostics): attribute storage reads to request origins
Port the diagnostics instrumentation from its experiment base onto main
without carrying the base branch's unrelated changes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(diagnostics): consolidate read context scopes
Keep caller and entry provenance in one task-local context, combine the object-read scope, and classify source paths without allocating.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(deps): patch profiling dependency for RUSTSEC-2026-0344
Vendor the published Pyroscope package with an object 0.40 dependency and
lock ruzstd 0.9.1. Preserve runtime sources and profiling features, add
bounded decoder regressions, and run the patched package in audit CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>
* fix(deps): correct vendored pyroscope documentation typos
Correct copied comments and changelog spelling for the repository typo
check, and describe these comment-only changes in the vendor provenance.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>
* fix(ecstore): refresh compact transition proof per test case
Install the short-lived homogeneous fleet proof immediately before each
kill-point transaction. Earlier cancellation and recovery work must not
age the next case's writer capability fixture.
Report an early transition result instead of waiting for an unreachable
kill point. Preserve cancellation, remote-owner and payload assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>
---------
Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Merge frozen main acd26afd8b and include
compatibility fixes and measured Darwin/Linux membership corrections
across six files.
Validation on candidate b4c3ab560a:
- All 20 local gates passed.
- Native smoke: 218 passed. Native full: 661 passed.
- Full passed on the same-head failed-only rerun; the initial failure
remains recorded.
- Gateway remains report-only: 38 passed, 180 failed, under the existing
exemption.
CI: https://github.com/rustfs/rustfs/actions/runs/38285788334
* feat(tls): reload external certificate snapshots coherently
Pin projected certificate generations, validate complete TLS material, coordinate server consumers, and invalidate outbound RPC caches coherently. Add externally managed TLS Secret support and rotation guidance.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(tls): publish server certificates through a fenced admin API
Add opt-in server-leaf publication with dedicated IAM actions, actual TLS
transport checks, immutable generations, durable captured-token CAS and
idempotent receipts. Preserve independent trust and client identity sources
and distinguish visible publication state from the full serving chain.
Validate 64 TLS tests, 41 TLS-runtime tests, the focused IAM policy test,
and eight live local API scenarios including least privilege and restart.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(admin): route certificate handler types through S3 facade
Use the existing admin S3 facade to keep direct s3s imports within the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(admin): include TLS certificate routes in route matrix
Account for the registered certificate-state GET and certificate publication POST without changing route behavior or weakening matrix assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(tls): reconcile startup trust before skipping reloads
Compare the initialized listener bundle with coherently published outbound material, retain startup trust and client identity when sources disappear, and cover independent startup rotations through real reload passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(tls): reuse pinned generations and prepared listener state
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(tls): preserve startup trust source requirements
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
T1.2 batch 3 (rustfs/backlog#2743): s3select-api, s3select-query and
the error items of protocols move off s3s by scripts/codemods, run with
--hold crates/protocols/src/sftp/errors.rs. That file stays on s3s: it
downcasts the backend error, whose runtime type is still s3s::S3Error.
- rustfs/src/protocols/client.rs: the two webdav session methods now
return rustfs_s3_types::S3Result and convert the s3s error with From.
- codemod: a child module that calls s3_error! through `use super::*`
gets its own `use rustfs_s3_types::s3_error;` once its parent imports
that macro by name; otherwise it is listed as GLOB.
- s3-types: a table test pins code, status and the s3s error the legacy
edge renders for every variant a moved call site names.
- s3_error_lines 1585 -> 1584 (a lifecycle comment reworded).
rustfs/backlog#2747 batch 3. `Body::from_stream` wraps a chunk stream of
unknown length over `StreamBody`/`BoxBody`: one data frame per chunk, the
first error ends the body, the size hint stays unknown, and dropping the
body drops the stream. Parity tests pin it to the s3s `StreamingBlob::wrap`
it replaces, at the `http_body` level and byte for byte through hyper.
The Swift handlers in rustfs-protocols build their responses as the owned
`Body`. `SwiftService` converts them into the wrapped S3 service's response
body type through `From<Body>`, so rustfs keeps serving `s3s::Body` at the
legacy edge unchanged. Request bodies stay Send-only (`UnsyncBoxBody`); the
two storage writers and the DLO/SLO segment streams get `Sync` from
`sync_wrapper::SyncStream`, which is what s3s's `http_body_unsync` mutex
provided before. The DTO and error uses of s3s in protocols are untouched
(T1.5 and T1.2).
Add rustfs_s3_types::Body, a BoxBody<Bytes, StdError> wrapper with
empty() and From<Bytes | Vec<u8> | String | &'static str>, and the
legacy From conversions to and from s3s::Body behind compat-s3s.
The signer is now generic over the request body: it only reads and
writes the request head, so it needs no body type and its callers keep
theirs. s3-client sends its requests with the owned Body, madmin signs a
bodiless head, and targets drops its two S3Request/S3Response header
wrappers in favour of extract_params_header.
rustfs-signer, rustfs-madmin and rustfs-targets no longer depend on
s3s; the s3s footprint baseline drops from 209 to 202 files.
Refs rustfs/backlog#2747, rustfs/backlog#2734
`rustfs-s3-types` now provides `S3ErrorCode` (generated by build.rs from
src/codes.txt, the grep-derived call surface, plus `Custom`), `S3Error`,
`S3Result`, `StdError` and the `s3_error` macro with the same call syntax
as s3s, so the T1.2 codemod is a `use` swap. Statuses are hand-written and
pinned to the s3s behaviour RustFS ships today by an oracle test that uses
s3s as a dev-dependency only. The `From` conversions for the legacy edge
live behind the off-by-default `compat-s3s` feature (the orphan rule
forbids them in rustfs/src/error.rs); `rustfs` enables it.
Refs rustfs/backlog#2742, rustfs/backlog#2734
* feat(targets): support Kafka SASL without TLS
Propagate the explicit Kafka transport mode through notify and audit targets while retaining TLS as the default.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): update RustFS Kafka crates to 1.4.0
Use the published Kafka clients after the SASL plaintext transport support release.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
overtrue