* fix(ci): include pagination regression in full E2E selection
* fix(deps): replace yanked yoke-derive release
* fix(scanner): expose pause backlog replica diagnostics (#8258)
* fix(scanner): expose pause backlog replica diagnostics
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(connect): stabilize runtime profile lease cancellation
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(connect): tolerate delayed schedule startup in CI
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): retry quota reads during usage warmup
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): avoid meta-bucket incarnation self-deadlock
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(test): use persisted incarnation in heal fixture
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(usage): reconcile stale counters after lifecycle expiration (#8108)
* fix(usage): reconcile stale counters after lifecycle expiration
* fix(usage): account lifecycle expiry during continuous writes
* test(usage): run lifecycle usage scenarios on one scanner store
* test(usage): use a Windows-representable pre-mutation offset
* fix(usage): harden expiry accounting recovery and quota checks
Borrow expiry receipt bucket names and avoid allocating a map key on cache hits. Cover cancelled receipts, durable snapshot recovery, and legacy quota admission after scanner confirmation. Use representable timestamp offsets in the quota regression.
Refs rustfs/backlog#2689
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(usage): recover stale persisted counts through the scanner
Seed incorrect complete usage for empty and retained-object buckets, then run the real scanner and publication consumer without further object mutations. Verify durable and admin usage over two cycles instead of writing a corrected snapshot in the test.
Refs rustfs/backlog#2689
Refs rustfs/backlog#2691
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): bound delimiter pagination fixture concurrency
The 120-second smoke timeout expired after 1018 of 1200 serial fixture PUTs, before LIST ran. Prepare the same objects with at most eight concurrent requests and await every PUT. Retain the timeout and strengthen exact prefix, KeyCount, empty Contents, and continuation-token assertions with phase diagnostics.
Refs rustfs/backlog#2689
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(upgrade): establish a persisted previous-release baseline
Seed the pinned previous-release cluster and restart it once with its data intact before replacing any node. Require every old writer to pass the strict readiness probe and preserve the seed through both mixed phases and the final current cluster. Keep InternalError fail-fast behavior and all existing compatibility deadlines and assertions.
Refs rustfs/backlog#2689
Refs rustfs/backlog#2384
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): bound cancellation metadata persistence waits
Use the system-bucket incarnation boundary now supplied by main PR #8268. Bound the three cancellation waits that previously hung during pool.bin persistence, retaining their remote-generation, target-cohort, and durable-state assertions.
Refs rustfs/backlog#2697
Refs rustfs/backlog#2689
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): retain startup and shutdown failure diagnostics
* fix(test): supply CPU workload for sampler regression
* fix(ci): locate security chain scripts in the workspace
* fix(usage): recover historical counters with generation fencing (#8273)
* fix(usage): recover historical counters during continued writes
Use newer converged scanner snapshots to reconcile stale absolute usage
baselines while preserving concurrent mutation and expiry receipt fences.
Cover durable publication, admin and quota reads, and legacy generations.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(usage): fence snapshots and move preserved cache entries
Apply the cached scanner generation floor before every reconciliation path
and retain it even when an older snapshot happens to match core counts.
Move preserved usage entries instead of cloning their histogram maps under
the cache lock, retaining expiry receipt identity and cancellation fences.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): verify checkpoint takeover and repair dispatch (#8275)
* test(scanner): cover checkpoint handoff and repair dispatch
Drive runtime budget expiry, partial-cycle persistence, leadership claims,
and stale checkpoint rejection between real disk-backed fixture scans.
Verify that a metadata repair beyond the first bounded prefix is saved in
the scanner ledger and dispatched by the MRF consumer.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci: isolate scanner fixtures and refresh full e2e membership
Reserve nextest capacity for the real-disk scanner publication and MRF
admission fixtures. Bind both platform membership checks to the reviewed
pagination deadline test added on main.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(scanner): consolidate checkpoint fixture lifecycle
Keep one durable control store across timeout and leadership transitions,
and inject generation advancement into the shared checkpoint scenario.
Check the actual saved metadata path so late-write rejection also proves
that existing checkpoint bytes remain intact.
Centralize MRF fixture isolation and reuse nextest process isolation when
the startup environment already satisfies the test contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(obs): distinguish allocator counters from live memory (#8274)
* fix(obs): distinguish allocator counters from live memory
Preserve count/counter semantics and mark requested-byte attribution unavailable when live statistics or sampling are missing. Document sustained multipart memory diagnosis.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(obs): parse allocator statistics from one node
Resolve each statistic before interpreting its shape, avoiding unsupported-field tree scans and mixing data across wrapper scopes. Preserve unavailable-statistics policy and add precedence regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): isolate late parity recovery from metadata hedges
Use the existing object-scoped hedge timer barrier in exact-count recovery fixtures. Preserve payload and total-read assertions and verify that the omitted parity disk is read only during late refresh.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(usage): combine identical snapshot retention branches
* fix(test): await HTTP sender readiness in Top RPC fixture
* [release/1.0.1] Gate multipart copy through write admission (#8284)
Gate multipart copy through write admission
Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* Gate multipart copy through write admission (#8283)
Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix: add UploadPart OOM validation guardrails (#8287)
* docs(release): validate candidates on release branch
* fix(scanner): validate checkpoints against global cycle fence (#8278)
## Related Issues
Related to rustfs/backlog#2701.
## Summary of Changes
Route scanner checkpoint cycle and leader validation through the global store while retaining the owning set for cache persistence, CAS revisions and publication admission.
## Verification
Two independent final-diff source reviews found no issues across correctness, concurrency and durability, test coverage, compatibility, performance and simplicity on head `8b8fe51d092090b053f552ae283960e2e306be33`. Root approval `5373624714` is bound to that exact head. Regression tests cover real two-pool routing, stale fences, post-save rejection and CAS conflicts; their reported local execution belongs to the PR author, not this merge operation. Current required CI remains pending, and this authorized admin squash does not establish CI or runtime acceptance.
## Impact
Restores checkpoint progress when global cycle and leader state differ from a set-scoped view. No format, retry, timeout, assertion or scanner-policy changes are introduced by this diff. The three prior main scanner failures remain unproved repaired.
## Additional Notes
Full validation must run on the resulting exact main revision. Reverting this patch restores the earlier set-scoped fence lookup and its checkpoint rejection behavior.
* fix(ci): restore E2E membership and pagination timeouts (#8281)
* ci: locate the auto-testing checkout for lanes that run evidence from a subdirectory (#8279)
## Related Issues
Follow-up to #8229.
## Summary of Changes
Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence.
## Verification
The exact PR head b66129ab9f passed one mechanical correctness and simplicity review, nine real-Git layout and provenance checks, and sixteen existing evidence/envelope tests. The baseline sibling layout failed with git exit 128; the corrected layout succeeded while revision mismatches and missing checkouts stayed rejected. Current PR checks are completed with successful or skipped conclusions, including the aggregate.
## Impact
Both lane and private-script revision checks remain intact. No time limits, assertions, production behavior, or evidence validation requirements change. The synthetic layout checks do not execute the actual scheduled security suite; integrated main CI and release acceptance remain separate gates.
## Additional Notes
Approved review 5374559393 is bound to the exact head above. Reverting the single-file change restores the previous checkout lookup.
* fix: add UploadPart OOM validation guardrails
Add a Docker validation harness for backlog#2704 so the ordinary UploadPart
low-concurrency memory workload can be reproduced with comparable case metadata,
process/cgroup sampling, TLS and metrics toggles, cache-env controls, and write
reclaim/direct-write experiments.
Warn when operators set the unrecognized RUSTFS_OBJECT_CACHE_* variables that
appeared in the reporter compose file. The variables are reported but remain
ignored, so startup does not silently change object data cache behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: AL <allan.bednarowski@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): pass fence store to checkpoint fixture
* fix(s3): queue bucket operations and restore strict Clippy checks (#8290)
* fix(s3): queue concurrent bucket creation and deletion
Keep eight active bucket transactions and bound admission waiting to 128 requests and 30 seconds. Preserve detached transaction ownership and return Retry-After with overload responses.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): restore strict Clippy compatibility on Rust 1.99
Use try_update without changing atomic ordering or overflow behavior. Keep
recursive storage futures boxed once at each frame and remove the redundant
async-recursion macro, including its non-recursive SQL planner use. Remove
needless closure borrows and orphaned dependency entries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore: refresh dependencies and atomic update APIs
Update workspace dependencies and the lockfile. Replace deprecated
atomic fetch_update aliases with try_update while preserving closures
and memory ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* improve
* fix(scanner): diagnose and verify pause backlog recovery (#8293)
* fix(scanner): diagnose and verify pause backlog recovery
Expose the retained replica snapshot and claimed membership in abnormal
admin status responses. Keep diagnostics off metrics updates and verify
single-pool recovery and conflicting-proof preservation across 24 sets.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(scanner): move replica snapshots into diagnostics
Consume the terminal admin read snapshot in a single state match and move
membership, revision, and error buffers into the response. Verify buffer
handoff and the unchanged JSON contract without altering ledger authority.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): wait for held legacy responsibility in replay test
* ci: remove Docker Hub description sync
* fix: emit NextPartNumberMarker only when ListParts is truncated
ListPartsInfo.next_part_number_marker was a non-optional usize that
defaulted to 0 and was only assigned when the response was truncated.
The S3 serializer then emitted it unconditionally as Some(0), causing
AWS SDK paginators to loop infinitely on part_number_marker=0 instead
of terminating.
Change the field to Option<usize> (None by default) and set it only
inside the is_truncated branch. The S3 output layer now uses
.and_then() so NextPartNumberMarker is absent when IsTruncated=false,
matching AWS S3 behavior.
Fixes#8208
(cherry picked from commit 44de803a38)
* fix(s3): honor sparse ListParts markers and verify termination
Resume part listings at the first part above the numeric marker, even
when that marker is absent. Use binary search over the sorted part
numbers and retain the existing exact-tail empty-slice path.
Add storage, XML, and real AWS SDK paginator regressions for empty and
terminal pages, sparse markers, and multipart completion integrity.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 673031eea1)
* fix(storage): publish delete rollback backups atomically
Stage rollback metadata outside the rollback directory and publish it only after the full write succeeds. A short write must not leave a backup that quorum rollback can rename over acknowledged version history.
Add an isolated real short-write regression and register the backported ListParts SDK test in the smoke and Linux full inventories.
* test(e2e): register paginator regression in Darwin inventory
* fix(release): install yq before Helm template checks
* chore(release): align installation references for 1.0.1
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Peder Bergan <pederbe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: AL <allan.bednarowski@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Chapman <touch65536@gmail.com>
Include helm/rustfs/README.md in the uploaded helm-package artifact so
publish-helm-package updates the root README.md in rustfs/helm.
The build-helm-package job copies helm/README.md into helm/rustfs/
before running helm package, but upload-artifact previously uploaded
only helm/rustfs/*.tgz. Because both files share the helm/rustfs/
parent directory, actions/upload-artifact places both the chart
archive and README.md at the artifact root, and download-artifact
extracts README.md into the root of rustfs/helm alongside the tarball.
* ci: replace ubuntu-latest runner with sm-standard-2 across workflows
* ci: keep scheduled-validation monitors on hosted runners
The freshness and watchdog jobs report stalled scheduled validations.
Running them on the same sm-standard-2 pool means a pool outage stalls
the monitors too, so nothing reports it.
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
actions/checkout writes its token into .git/config as an http extraheader,
where it stays for the rest of the job. That matters more here than usual:
pull_request jobs run on self-hosted runners and execute the PR's own build.rs,
proc-macros and tests, any of which can read that file. Test and Lint holds
actions: write on top of that, so its token can cancel runs and delete the
Actions caches the whole pipeline now depends on — it was given
persist-credentials: false when that permission was added, and this extends the
same treatment to the other 45 checkouts.
Two are exempt because the token IS the credential the job needs.
helm-package's publish job pushes to rustfs/helm with it; clearing it would
break chart publishing. nix-flake-update is exempt pending verification: it
passes FLAKE_UPDATE_TOKEN to create-pull-request directly rather than reusing
.git/config, so it very likely does not need persistence, but that is unproven
and a broken weekly bot is not worth the guess. Both carry a
persist-credentials-exempt comment saying which.
scripts/security/check_persist_credentials.sh requires every checkout to either
clear its credentials or carry that comment, so the decision stays visible in
review rather than being an omission nobody notices.
Refs: rustfs/backlog#1598, rustfs/backlog#1602
Three hardening changes with no effect on what any workflow produces.
Declare timeout-minutes on the 25 jobs that lacked it. GitHub's default is 360
minutes, and this repository has a history of runners stalling intermittently
(#5394) plus a measured 9m57s plain `git checkout` under node-level I/O
contention, so one wedged job could hold a runner for six hours out of a pool of
roughly 15-21. Budgets follow what the jobs actually do: 10 minutes for
echo-only and guard-script jobs, 30 for anything calling the GitHub API,
uploading release assets or pushing over the network.
scripts/security/check_job_timeouts.sh keeps it that way, checking only jobs
that declare runs-on so reusable-workflow callers are not flagged.
Pass workflow inputs and workflow_run fields through env instead of `${{ }}`
interpolation in run blocks. A git ref name may contain `$(...)` — any string
without a space is a legal tag — and interpolation pastes it into the script
where bash evaluates it. The worst instance was helm-package's final commit
message: it is built from the triggering tag name inside the job that holds the
cross-repository push token with rustfs/helm already checked out. Also converted
in build.yml, docker.yml and performance-ab.yml; the last is currently disabled,
but a disabled workflow can be re-enabled. Not touched: helm-package's
`contains(head_branch, '.')` tag test, since GitHub expressions have no regex
and this repository's tags carry no `v` prefix, so rewriting the condition would
change which builds publish a chart.
Give audit.yml a scheduled-failure alert and run it daily. A scheduled
cargo-deny failure usually means the dependency tree just matched a newly
published RustSec advisory — the most important signal this workflow produces,
and until now it was visible only to whoever happened to open the Actions tab.
coverage.yml and e2e-replication-nightly.yml already use this ci-8 mechanism.
The cron moves from weekly to daily so a new advisory against an unchanged tree
surfaces within a day instead of seven; the check list is untouched, since
splitting it into a light daily run and a weekly full run would create runs
where sources, bans and licenses go unverified.
Refs: rustfs/backlog#1598, rustfs/backlog#1602
Chris