ListPartsInfo.next_part_number_marker was a non-optional usize that
defaulted to 0 and was only assigned when the response was truncated.
The S3 serializer then emitted it unconditionally as Some(0), causing
AWS SDK paginators to loop infinitely on part_number_marker=0 instead
of terminating.
Change the field to Option<usize> (None by default) and set it only
inside the is_truncated branch. The S3 output layer now uses
.and_then() so NextPartNumberMarker is absent when IsTruncated=false,
matching AWS S3 behavior.
Fixes#8208
(cherry picked from commit 44de803a38)
* fix(scanner): diagnose and verify pause backlog recovery
Expose the retained replica snapshot and claimed membership in abnormal
admin status responses. Keep diagnostics off metrics updates and verify
single-pool recovery and conflicting-proof preservation across 24 sets.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(scanner): move replica snapshots into diagnostics
Consume the terminal admin read snapshot in a single state match and move
membership, revision, and error buffers into the response. Verify buffer
handoff and the unchanged JSON contract without altering ledger authority.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Integrate the updated release/1.0.1 branch before pushing the
dependency and atomic API maintenance changes. Preserve the upstream
removal of async-recursion.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Update workspace dependencies and the lockfile. Replace deprecated
atomic fetch_update aliases with try_update while preserving closures
and memory ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* docs(release): validate candidates on release branch
* fix(scanner): validate checkpoints against global cycle fence (#8278)
## Related Issues
Related to rustfs/backlog#2701.
## Summary of Changes
Route scanner checkpoint cycle and leader validation through the global store while retaining the owning set for cache persistence, CAS revisions and publication admission.
## Verification
Two independent final-diff source reviews found no issues across correctness, concurrency and durability, test coverage, compatibility, performance and simplicity on head `8b8fe51d092090b053f552ae283960e2e306be33`. Root approval `5373624714` is bound to that exact head. Regression tests cover real two-pool routing, stale fences, post-save rejection and CAS conflicts; their reported local execution belongs to the PR author, not this merge operation. Current required CI remains pending, and this authorized admin squash does not establish CI or runtime acceptance.
## Impact
Restores checkpoint progress when global cycle and leader state differ from a set-scoped view. No format, retry, timeout, assertion or scanner-policy changes are introduced by this diff. The three prior main scanner failures remain unproved repaired.
## Additional Notes
Full validation must run on the resulting exact main revision. Reverting this patch restores the earlier set-scoped fence lookup and its checkpoint rejection behavior.
* fix(ci): restore E2E membership and pagination timeouts (#8281)
* ci: locate the auto-testing checkout for lanes that run evidence from a subdirectory (#8279)
## Related Issues
Follow-up to #8229.
## Summary of Changes
Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence.
## Verification
The exact PR head b66129ab9f passed one mechanical correctness and simplicity review, nine real-Git layout and provenance checks, and sixteen existing evidence/envelope tests. The baseline sibling layout failed with git exit 128; the corrected layout succeeded while revision mismatches and missing checkouts stayed rejected. Current PR checks are completed with successful or skipped conclusions, including the aggregate.
## Impact
Both lane and private-script revision checks remain intact. No time limits, assertions, production behavior, or evidence validation requirements change. The synthetic layout checks do not execute the actual scheduled security suite; integrated main CI and release acceptance remain separate gates.
## Additional Notes
Approved review 5374559393 is bound to the exact head above. Reverting the single-file change restores the previous checkout lookup.
* fix: add UploadPart OOM validation guardrails
Add a Docker validation harness for backlog#2704 so the ordinary UploadPart
low-concurrency memory workload can be reproduced with comparable case metadata,
process/cgroup sampling, TLS and metrics toggles, cache-env controls, and write
reclaim/direct-write experiments.
Warn when operators set the unrecognized RUSTFS_OBJECT_CACHE_* variables that
appeared in the reporter compose file. The variables are reported but remain
ignored, so startup does not silently change object data cache behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: AL <allan.bednarowski@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Gate multipart copy through write admission
Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(obs): distinguish allocator counters from live memory
Preserve count/counter semantics and mark requested-byte attribution unavailable when live statistics or sampling are missing. Document sustained multipart memory diagnosis.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(obs): parse allocator statistics from one node
Resolve each statistic before interpreting its shape, avoiding unsupported-field tree scans and mixing data across wrapper scopes. Preserve unavailable-statistics policy and add precedence regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): isolate late parity recovery from metadata hedges
Use the existing object-scoped hedge timer barrier in exact-count recovery fixtures. Preserve payload and total-read assertions and verify that the omitted parity disk is read only during late refresh.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): cover checkpoint handoff and repair dispatch
Drive runtime budget expiry, partial-cycle persistence, leadership claims,
and stale checkpoint rejection between real disk-backed fixture scans.
Verify that a metadata repair beyond the first bounded prefix is saved in
the scanner ledger and dispatched by the MRF consumer.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci: isolate scanner fixtures and refresh full e2e membership
Reserve nextest capacity for the real-disk scanner publication and MRF
admission fixtures. Bind both platform membership checks to the reviewed
pagination deadline test added on main.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(scanner): consolidate checkpoint fixture lifecycle
Keep one durable control store across timeout and leadership transitions,
and inject generation advancement into the shared checkpoint scenario.
Check the actual saved metadata path so late-write rejection also proves
that existing checkpoint bytes remain intact.
Centralize MRF fixture isolation and reuse nextest process isolation when
the startup environment already satisfies the test contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(usage): recover historical counters during continued writes
Use newer converged scanner snapshots to reconcile stale absolute usage
baselines while preserving concurrent mutation and expiry receipt fences.
Cover durable publication, admin and quota reads, and legacy generations.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(usage): fence snapshots and move preserved cache entries
Apply the cached scanner generation floor before every reconciliation path
and retain it even when an older snapshot happens to match core counts.
Move preserved usage entries instead of cloning their histogram maps under
the cache lock, retaining expiry receipt identity and cancellation fences.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(usage): reconcile stale counters after lifecycle expiration
* fix(usage): account lifecycle expiry during continuous writes
* test(usage): run lifecycle usage scenarios on one scanner store
* test(usage): use a Windows-representable pre-mutation offset
* fix(usage): harden expiry accounting recovery and quota checks
Borrow expiry receipt bucket names and avoid allocating a map key on cache hits. Cover cancelled receipts, durable snapshot recovery, and legacy quota admission after scanner confirmation. Use representable timestamp offsets in the quota regression.
Refs rustfs/backlog#2689
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(usage): recover stale persisted counts through the scanner
Seed incorrect complete usage for empty and retained-object buckets, then run the real scanner and publication consumer without further object mutations. Verify durable and admin usage over two cycles instead of writing a corrected snapshot in the test.
Refs rustfs/backlog#2689
Refs rustfs/backlog#2691
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): bound delimiter pagination fixture concurrency
The 120-second smoke timeout expired after 1018 of 1200 serial fixture PUTs, before LIST ran. Prepare the same objects with at most eight concurrent requests and await every PUT. Retain the timeout and strengthen exact prefix, KeyCount, empty Contents, and continuation-token assertions with phase diagnostics.
Refs rustfs/backlog#2689
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(upgrade): establish a persisted previous-release baseline
Seed the pinned previous-release cluster and restart it once with its data intact before replacing any node. Require every old writer to pass the strict readiness probe and preserve the seed through both mixed phases and the final current cluster. Keep InternalError fail-fast behavior and all existing compatibility deadlines and assertions.
Refs rustfs/backlog#2689
Refs rustfs/backlog#2384
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): bound cancellation metadata persistence waits
Use the system-bucket incarnation boundary now supplied by main PR #8268. Bound the three cancellation waits that previously hung during pool.bin persistence, retaining their remote-generation, target-cohort, and durable-state assertions.
Refs rustfs/backlog#2697
Refs rustfs/backlog#2689
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(test): establish a writable previous-release upgrade baseline
* fix(ci): reserve capacity for durable admin fixtures
* fix(ci): group durable IAM state fixtures by resource needs
* ci: run E2E doctests with the E2E dependency graph
* fix(ci): separate fixture startup from transport deadlines
* fix(ci): bound pagination after seeding and revisit restored copies
* fix(ci): make filesystem fixture timing deterministic
* fix(ecstore): avoid metadata lock reentry during internal mutations
* test: align recovery fixtures with durable ownership contracts
## Related Issues
N/A
## Summary of Changes
Reject system-bucket incarnation lookups before entering the pool metadata owner. Keep each selected scanner backlog publication cohort in one task so waiter cancellation cannot abandon its remaining serialized conditional writes. Bind repair and replay fixtures to persisted bucket incarnations.
## Verification
Head f6d44603fc received an approval from houseme in review 5365588011. This merge does not add a local runtime validation claim. Full main CI remains a separate publication gate.
## Impact
System metadata writes avoid recursive pool locking. Publication retains per-replica conditional writes and stops a cancelled caller from advancing to its next publication phase. Fixture changes supply the identities required by existing repair admission rules.
## Additional Notes
Reverting this change restores the previous behavior.
## Related Issues
Related to rustfs/backlog#2688.
## Summary of Changes
Reclaim metadata-less orphan directory trees after a complete, first-page, empty recursive root listing in an authoritative never-versioned bucket. Both listing implementations use the same fail-closed cleanup decision.
## Verification
Head b4a9120173 received an approval from loverustfs in review 5365460152. This merge does not add a local runtime validation claim. Full main CI remains a separate publication gate.
## Impact
Empty recursive root listings can remove otherwise unaddressable physical residue. Versioned buckets and populated listings remain excluded; unreadable disks, object metadata, unknown files, and uncommitted data prevent cleanup.
## Additional Notes
Reverting this change restores the previous orphan-directory behavior.
Include helm/rustfs/README.md in the uploaded helm-package artifact so
publish-helm-package updates the root README.md in rustfs/helm.
The build-helm-package job copies helm/README.md into helm/rustfs/
before running helm package, but upload-artifact previously uploaded
only helm/rustfs/*.tgz. Because both files share the helm/rustfs/
parent directory, actions/upload-artifact places both the chart
archive and README.md at the artifact root, and download-artifact
extracts README.md into the root of rustfs/helm alongside the tarball.
## Related Issues
Resolves the corrected lifecycle fixture findings in PR #8254.
## Summary of Changes
Preserve separate incarnation-bound durable MRF responsibilities and their lifecycle audit records, and align replay fixtures with their checkpoint identities.
## Verification
Two independent source reviews and changed-delta reviews are complete. The final review approved 607a0b9bb0 with no remaining supported findings. Local runtime claims were not independently reproduced for this pull request.
## Impact
Keeps storage generation fences and retained responsibility semantics. Test-capacity reservations preserve existing deadlines and assertions.
## Additional Notes
Squash merge of the currently approved fix under the authorized CI-bypass exception. Main CI and release acceptance remain required.
## Related Issues
Follow-up to #8233.
## Summary of Changes
Registration runtime fixtures timed out in the workspace CI lane while the same cases passed in the feature lanes. Reserve nextest capacity for the exact `connect_registration` binary, as already done for related inventory and drive fixtures. Keep its existing deadlines, internal concurrency, assertions and zero-retry policy; report the last watch status on failure.
Rolling upgrades could pass `ListBuckets` readiness while restarted peers still lacked write quorum. Before each mixed-version phase, probe writes through every node outside the asserted workload prefixes. A shared 30-second deadline includes requests and sleeps; only HTTP 503 with `ServiceUnavailable` is retryable, with SDK retries disabled for these probes. The actual compatibility writes, reads, multipart operations and listing assertions remain unchanged.
Add four fast regression tests to the existing PR smoke profile, with matching exclusion from the full profile. No new workflow or job is introduced.
## Verification
- `cargo nextest run --locked --profile ci -p rustfs --lib --test connect_registration --test-threads 4 -E 'binary(/^connect_registration$/) | test(=connect::diagnostics::trace_runtime::tests::local_runtime_rejects_non_private_state)' --no-tests fail --status-level pass --final-status-level fail` passed 23/23 twice in 5.088s and 5.097s: 22 macOS registration tests plus one unrelated control. JUnit intervals confirm capacity reservation; no retries or test-process leaks occurred. The Linux-only inventory case remains for CI.
- `cargo nextest run -p e2e_test --lib --profile ci -E 'test(upgrade_write_readiness_tests)' --no-fail-fast --no-tests fail` passed 4/4 twice in 1.068s and 1.072s, with zero retries. Regressions cover metadata readiness followed by write unavailability, recovery through every writer, immediate permanent-error failure despite an incoming SDK retry configuration, and deadlines for repeated 503s and stalled requests. The original failure is recorded in [the mixed-version upgrade job](https://github.com/rustfs/rustfs/actions/runs/36569700716/job/109415806473).
- `cargo fmt --all --check`, `git diff --check`, `python3 scripts/check_test_wiring.py` and compiled smoke/full membership checks passed. Smoke membership changes from 188 to 192 by adding exactly these four tests; full membership is unchanged. The expected Linux smoke digest was derived from the actual prior Linux listing plus those four platform-independent additions and still requires confirmation by this PR's CI.
Local verification covers the exact source committed in `9b2ea836317ed035a91d1fc9fcf725f70c3098e2` on main `380e98a42cb4fcd0994fed79b30c2c7605deb0bc`. An independent final-diff correctness and reliability review found no findings. Fresh Linux workspace and real mixed-version upgrade runs are required before treating the remediation as fully verified; local fake-target tests do not establish that result.
## Impact
Test scheduling and readiness only; no production behavior, API, dependency, test deadline or compatibility assertion changes. Reserving capacity serializes registration fixture processes within a nextest run. The bounded readiness probes may add startup time while peer write health converges; permanent errors still fail immediately.
## Additional Notes
Rollback by reverting this PR. Existing CI restructuring from #8233 is independent of these follow-up fixes.
* fix(ci): reduce duplicate work and preserve reliable test failures
* fix(ci): retain protocol evidence and repair stale test fixtures
* test(connect): honor parent deadline during API fixture readiness
* fix(ci): reserve IO capacity for state writer proofs
* test(connect): align RPC fixtures with service capture contracts
* test(connect): cover pinned service capture failures
* feat(connect): capture offline network performance in server
* fix(connect): use storage facade in RPC test
* fix(connect): keep RPC test body behind storage facade
Chapman