* test(e2e): use the resumed decommission completion budget
* test(heal): exercise checkpoints written by rc.5
* test(heal): route upgrade fixtures through the storage facade
* fix(deps): update ctutils to address constant-time advisory
* test(lifecycle): gate recovery I/O while advancing test time
* test(e2e): add scanner/heal topology protection matrix
Add the S00 protection baseline for the Scanner/Heal program
(rustfs/backlog#2873, program rustfs/backlog#2872): seven
scanner_heal_topology_ cases in the distributed e2e lane that pin pool
expansion, rebalance and decommission behavior before the runtime changes.
The fixture seeds a deterministic inventory (empty, inline, external,
multipart, special and nested keys; a versioned bucket with an explicit
null version, three history versions and a delete marker per key; an
Object Lock bucket with legal hold and COMPLIANCE retention), records
every version with its requested and listed version id, length and
SHA-256, verifies the manifest through every node, reads physical
placement from xl.meta on the pool drives, and reads the strict
durability mode back from the admin API and the node logs.
Fault cases hold the mover at proven windows through file barriers that
compile only with the e2e-test-hooks feature: before the multipart target
commit and before source cleanup (shared by decommission and rebalance),
plus the existing pool-metadata before_publish cutpoint. An unreached
window fails the case. DistCluster gains capture_logs_under so a case can
read node startup events without a global environment variable.
docs/testing/scanner-heal-topology-gates.md maps the gates to the tests
and records the timeout basis.
* test(e2e): keep the barrier failure message static
The error(format!) ratchet rejects new formatted io::Error::other call
sites in ecstore. The reached observation already records the stage and
object, so the injected failure needs no per-call detail.
* test(e2e): require e2e-test-hooks for topology fault cases
The four scanner_heal_topology_ fault cases inject at server barriers that
exist only with the e2e-test-hooks feature. Against a default server they
waited for the barrier timeout and reported it as not reached. Check the
verified binary's features first so the missing prerequisite fails
immediately and is named.
* ci(e2e): build the distributed lane with e2e-test-hooks
Every other e2e lane builds and runs the fault-test hooks server. The
distributed lane alone used the default server, so the topology fault
cases could never reach their barriers there. Build and run with the same
feature so the receipt matches, and update the documented local commands.
The upgrade cases spawn the pinned previous release directly and do not
compare feature sets.
* test(e2e): keep topology artifacts under RUSTFS_E2E_LOG_DIR
The topology fixture captured node logs, barrier observations and evidence
in a temp dir removed on drop, so the distributed workflow uploaded nothing
for these cases. Fall back to RUSTFS_E2E_LOG_DIR, which the workflow
uploads, when no explicit evidence directory is set.
* test(e2e): remove retired pools before restart; race the pre-ACK start
The queue case restarted with decommissioned pools 0 and 1 still on the
command line, which the server rightly refuses. Add
remove_single_node_pools and restart without them, then expect the
remaining pools renumbered to 0 and 1.
The pre-ACK case now races the start request against the barrier. A start
that returns first, or stays in flight past the documented 60 s budget, is
reported as such instead of as an unreached barrier, and the evidence
records the reach latency.
capture_logs_under now overrides a log path assigned from
RUSTFS_E2E_LOG_DIR, so the seed node logs into the case directory too.
* test(e2e): bound fault-window waits by the movement budget
On Linux CI a busy target capacity gate delayed the first fault object by
about two minutes, past the 120 s wait borrowed from the server's release
poll. Wait for a movement window within the 240 s movement budget instead,
and fail at once when the decommission or rebalance ends without entering
the window, so an uninjected fault is never counted.
* test(e2e): bound status probes while waiting for a fault window
The admin decommission status request can wait on a mover held at a test
barrier until the barrier's own 120 s release timeout. The window wait
issued that request each loop, so a reached barrier went unnoticed until
the server timed it out and failed the decommission, which surfaced as a
false resume failure. Bound each status probe to 5 s and treat a slow probe
as still running.
* test(e2e): retry a busy decommission start
A decommission start that meets a busy movement gate can answer a
retryable 503 instead of waiting. The harness treated any non-proof
failure as fatal. Share one start helper that retries 503 within the
case's movement budget and the fleet-proof 500 for 45 s, and keeps every
other status fatal. The pre-ACK case retries 503 within its window budget.
ci(pool): finalize decommission removal and verify surviving pools (rustfs#8726)
Select suite step 10 together with step 9 and pass
--finalize-decommission so the nightly exercises the post-removal
survivor path: pool ids must stay unique, bucket metadata writes must
keep succeeding across a restart, and no node may log 'pool metadata
layout changed'. Step 10 self-skips when decommission is disabled.
Co-authored-by: Hauser <housemecn@gmail.com>
Apply the placement rule in .github/workflows/README.md to the 22 jobs
still on sm-standard-2 that only call APIs, read workflow metadata, print
summaries or scan published images, including cla.yml, the busiest
workflow. Moving the ODM cloud-source lane also stops it restoring the
ci-odm-interop key across images, since its writer already runs on
ubuntu-latest; targets-live gets a -hosted cache key for the same reason.
Clippy ran only after merge, so lint regressions merged freely and then
failed every main run until fixed (10 of 12 main runs on 2026-10-17/18).
Run the workspace Clippy step on every event, warm its check-mode
artifacts in cache-warm with the identical invocation so pull requests
re-check only what they touch, and pin the unconditional step in the
ci_gate contract and its self-test.
The first pull-request run at CARGO_BUILD_JOBS=3 that met a contended
node (39193827281) hit the 75-minute nextest timeout before any test
started. Its sampler shows cargo spinning for 70 minutes with the pod's
io.pressure "full" at 86-88% and cpu.pressure at zero, with node-level
IO pressure already at 36% when the step began: three concurrent
workspace-scale compile units saturate the pod's shared storage, the
mechanism the original #5394 mitigation named. Keep 3 on main pushes and
dispatches, where three runs succeeded, and 2 on pull requests, in both
the nextest and the serial ILM step; pin the ILM budget guard back to
that expression and record the evidence in the step comment.
#8682 replaced the cold `cargo check -p rustfs --release` with a
`cargo rustc ... -C debug-assertions=off` on the rustfs lib alone. That
leaves the dependencies on the dev-profile cfg, so the lib also fails on
items gated on one side of cfg(debug_assertions) only (E0599 on
Filenames::Embedded), and the step's "fails solely at the compile_error"
assertion can never hold; the lane has been red on main and every pull
request since. Restore the whole-graph release check, drop the two-job
limit that only a linking step needed, and record why the single-crate
variant is unsound.
Raise CARGO_BUILD_JOBS to 3 on every event in the workspace test lane and
record the closed backlog#1601 experiment: the test-profile compile took
28m21s at 2 on a pull request and 16-23 min at 3 across three successful
main runs.
Merge the two Connect boundary jobs into connect-feature-boundaries. The
offline enrollment gate no longer exports a fresh CARGO_TARGET_DIR, so it
rebuilds only the rustfs crate on top of the restored ci-dev cache; the E2E
root is isolated by the feature gate and the dedicated binary name, not by
the target directory. The short-credential release guard is now
`cargo rustc -p rustfs --lib --features connect-e2e-short-credentials --
-C debug-assertions=off`, which trips the same cfg(not(debug_assertions))
compile_error against cached dev-profile dependencies instead of a cold
release check of the whole graph. Every assertion of both lanes is kept.
Update ci_gate.py and the ci-gates.md PR matrix for the merged job.
* ci: move compile-free and prebuilt-binary lanes to hosted runners
Standard GitHub-hosted runners are free for this public repository, while
every sm-standard minute is paid self-hosted capacity. Move the jobs that
never compile (quick checks, typos, scope selection, the required-checks
aggregate, audit, closed-PR cancellation) and the two s3-tests lanes that
only run the prebuilt debug binary to ubuntu-latest, drop the wget install
the custom image needed for typos, and move the upgrade-compatibility
workflow, whose build shape already runs on hosted runners in
e2e-distributed, with renamed cache keys so the hosted and self-hosted
images never share a rust-cache entry. Gate the package summary job on
resolve having run, so development builds stop starting a pod to report a
skipped job.
* docs(ci): document runner billing and placement rules
Record in .github/workflows/README.md which runner labels GitHub bills
(none for this public repository), which ones RustFS pays for, the rule for
when a job may leave ubuntu-latest, the rust-cache single-writer and
same-image constraints, the patterns that have leaked self-hosted minutes,
and a checklist for new or moved jobs. Point to it from .github/AGENTS.md,
docs/testing/ci-gates.md and the ci.yml placement comment so the rule has a
single home.
* fix(utils): NUL-terminate and extend paths passed to Win32 volume APIs
`get_volume_name` returned the volume root truncated at its NUL and the
result was then passed to `GetVolumeInformationW` as a PCWSTR, an
out-of-bounds read. The disk-space and volume APIs also received the
caller's raw path, so they failed beyond MAX_PATH, and a UNC share root
lacked the trailing backslash these APIs require.
- Wrap every Win32 path in a NUL-terminated `WidePath` that rejects
interior NULs; volume roots returned by `GetVolumePathNameW` keep
their terminator.
- Convert inputs to the extended-length `\\?\` / `\\?\UNC\` form after
`GetFullPathNameW` normalization, and give directory queries a
trailing backslash.
- Query `GetDiskFreeSpaceW` and `GetVolumeInformationW` on the volume
root as documented; size buffers to the documented MAX_PATH + 1 or
the input length.
- Report Win32 error codes instead of raw HRESULTs so `ErrorKind`
matches `std::fs`.
- Keep volume ids in their conventional `C:\` / `\\server\share\` form.
Add Windows-only `os::` tests for long directories, UNC share roots and
non-empty fstype, and run them in the Windows filesystem workflow.
* fix(utils): report Windows network volumes to the fs guard
Windows names the server's filesystem (usually NTFS) for SMB shares, so
the startup unsupported-filesystem guard, which matches fstype names
such as cifs or smb2, never fired for UNC paths or mapped drives.
- Add `DiskInfo::remote`. On Windows it is set when the directory handle
answers `FileRemoteProtocolInfo` (only network redirectors do, and the
handle follows symlinks and junctions to their target) or the volume
root's `GetDriveTypeW` is `DRIVE_REMOTE`. Unix keeps it false because
the client fstype already names network mounts.
- Treat remote volumes as unsupported in the startup guard and label
them "<fstype> on network share".
- Cover UNC shares, paths beyond MAX_PATH below a share, a directory
symlink to a share and a `net use` mapped drive with Windows-only
`os::` tests that skip when `\\localhost\<drive>$` is unreachable;
the Windows filesystem workflow maps the drive before running them.
* chore(deps): replace yanked inout 0.2.3 with 0.2.4
cargo-deny's advisories check rejects the yanked inout 0.2.3 that main's lockfile pins, failing every open PR. Lockfile-only patch bump.
* fix(utils): detect network targets from the final NT path
On the Windows runner, a directory symlink to \\localhost\C$ was reported as local: FileRemoteProtocolInfo did not identify it, and the volume root of the link is a fixed drive.
Also treat a handle whose final NT path (GetFinalPathNameByHandleW with VOLUME_NAME_NT) lies under \Device\Mup\ as remote; the Multiple UNC Provider fronts every UNC and mapped-drive target. The tests now assert the handle probe on its own for local and share paths and report both signals when it disagrees.
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ecstore): share metadata cache identity across Windows key aliases
Windows disks treat `\` as a path separator and `path_join_buf` drops the
empty segments that leaves, so `a\b`, `a\\b`, `\a\b` and `a\b\` all
address the xl.meta of `a/b`. The GetObject metadata cache keyed and
hashed the raw spelling, so DELETE `a\b` retired nothing cached for
`a/b`: GET/HEAD `a/b` kept returning the deleted object for up to the
2s cache TTL.
Fold the object name into the same segment form the disk resolves to
before computing the cache hash, key and lookup key. Off Windows the
name stays verbatim and borrowed, so `\` remains an ordinary key byte.
* fix(ecstore): serialize Windows backslash key aliases on one namespace lock
Windows disks treat `\` as a path separator and drop the empty segments
it leaves, so `a\b`, `a\\b`, `\a\b` and `a\b\` all address the xl.meta
of `a/b`. Namespace locks keyed the raw spelling, so writes, deletes and
heals through different spellings ran concurrently on one xl.meta and a
read-modify-write in rename_data could drop a version.
Fold the object name into its disk identity (rustfs_utils::path::
object_name_disk_identity, a no-op off Windows) wherever a namespace
lock key is formed:
- SetDisks::new_ns_lock, which every object lock path goes through;
- the batch delete lock request and its dedup/lookups, which build
ObjectKeys directly.
Two consumers compared lock identities by spelling and follow along:
- WriteCommitGuard records the folded key from the lock resource, so
matching a borrowed guard now folds the target name too;
- CopyObject holds the source read guard until EOF on streaming reads.
`a\b\` -> `a/b` is not a same-object copy by path_join_buf, but shares
one lock, so it now takes the self-copy lock instead of waiting on
itself until the lock timeout.
Distributed lock keys are folded by the requesting node only; lock
servers compare them verbatim.
* refactor(ecstore): key the metadata cache by the shared disk identity
The metadata cache fix carried its own copy of the Windows backslash
folding rule. Use rustfs_utils::path::object_name_disk_identity, which
namespace locks already use, so cache and lock identities cannot drift.
* ci(windows): run filesystem tests when key identity rules change
The Windows backslash folding for object names lives in utils path.rs
and is consumed by the namespace lock and write commit guard; run the
windows_ tests when any of them changes.
* fix(ecstore): keep Windows disk roots out of verbatim form
The disk root comes from GetFinalPathNameByHandleW, which returns a
verbatim path. `simplified` only strips `\\?\C:` below MAX_PATH, so a
data directory past 260 characters (or one reached through a UNC share)
kept the prefix. Disk IO extends the root by joining `/`-separated
relative paths as strings (`read_raw` builds `<object>/xl.meta` with
`path_join`), and inside a verbatim path `/` is not a separator: the
first metadata read, pool.bin.identity, failed with ERROR_INVALID_NAME
until startup gave up.
Add `rustfs_utils::non_verbatim`, which strips the disk and UNC verbatim
prefixes without a length limit while still keeping paths whose
components Win32 would rewrite, and use it for the publication root.
std re-adds the extended-length prefix for long paths.
Cover a root past MAX_PATH with a disk-level read/write test and run all
`dunce::` tests in the Windows filesystem workflow.
* fix(ecstore): query Win32 volume APIs with extended long disk roots
A disk root past the legacy limit used to reach `rustfs_utils::os` in
verbatim form, which the Win32 volume APIs accept. Now that the root
stays non-verbatim, GetDiskFreeSpaceExW rejected it with
ERROR_PATH_NOT_FOUND and LocalDisk::new failed in the Windows filesystem
workflow. Pass long roots to `get_info` and `get_physical_device_ids` in
extended-length form; shorter roots stay unchanged so reported volume
identities keep their `C:\` form.
On Windows, URL endpoints such as http://host:9000/C:/data{1...4} (the only
form multi-node and pool-expansion layouts accept) produced the disk path
`/C:/data1`: `try_from` stripped the synthetic slash, but `Url::set_path`
put it back, and `get_file_path` only removed it for `file` endpoints.
Windows resolves `/C:/data1` as an invalid `C:` component under the current
drive, so disk validation and `resolve_local_disk_root` failed.
- `get_file_path` now restores `/X:/...` to `X:/...` for every scheme on
Windows; URL paths without a drive (`/export1`, `/C:foo`) are unchanged.
- UNC shares (`\\nas\share\d1`, `//nas/share/d1`, `\\?\UNC\...`) are rejected
with an explicit error instead of becoming a file URL whose host is the
share server and whose path drops the share name.
- A drive root (`http://host:9000/D:/`) is accepted in URL form, matching
the path-style `D:\` endpoint that was already accepted.
- The "distinct physical disks" error explains on Windows that directories
on one volume share a device identity, alongside the existing
RUSTFS_UNSAFE_BYPASS_DISK_CHECK hint.
- The Windows filesystem lane now runs on layout changes and runs the
layout::endpoint:: tests; new Windows tests carry the windows_ prefix.
- Document Windows endpoint forms, the same-volume check, and the SNSD
expansion limit in cluster-lifecycle-operations.md.
`run_process` called `block_on(async_main())` on the native main thread,
so the whole startup chain ran on whatever stack the platform hands the
main thread: 8 MiB from `ulimit -s` on Linux and macOS, but the 1 MiB PE
default on Windows. A Windows debug build overflowed before it listened.
`RUST_MIN_STACK` does not reach the native main thread.
Run the runtime on a `rustfs-main` thread with an explicit 8 MiB stack,
keeping the budget Linux and macOS already had and making it independent
of the linker, toolchain (msvc/gnu) and `ulimit`. Panics are re-raised
on the main thread so the exit status is unchanged.
Measured on macOS arm64 by lowering the main-thread stack with
`ulimit -s` before this change: debug startup overflows at 1792 KiB and
reaches /health/ready at 2048 KiB; release overflows at 512 KiB and is
ready at 768 KiB. After it, debug starts with a 256 KiB main thread.
Add a Windows `Startup Smoke` job that builds the debug server, runs
`--version`, and waits for /health/ready on a single volume.
* fix(ecstore): admit CopyObject destinations like PutObject
Object-key platform checks were split across call sites with gaps:
CopyObject only ran check_bucket_and_object_names on its destination and
then reached the pools' put_object directly, so it admitted keys PutObject
rejects (over 1024 bytes, leading slash, and on Windows reserved characters
or unaddressable segments, the "writes but cannot be read" case of #3449).
- rustfs-utils: expose check_windows_file_name as the single per-segment
Win32 rule set; dunce's simplified() reuses it. Reserved device names now
include COM/LPT with superscript digits and CONIN$/CONOUT$.
- ecstore: replace the duplicated reserved-name table with that function;
Windows also rejects control characters 0x01-0x1F (NUL keeps its
platform-independent ObjectNameInvalid answer).
- ecstore: add check_new_object_args as the shared admission entry for
PutObject, cross-key CopyObject destinations and CreateMultipartUpload.
In-place copies (metadata replace, restore) keep the copy-only check so
existing keys stay rewritable. CreateMultipartUpload now also rejects
over-NAME_MAX segments up front instead of failing at completion.
- CI: run bucket::utils and dunce tests in the Windows filesystem job.
* fix(ecstore): reject new backslash keys on Windows hosts
A Windows disk treats '\' as a path separator, so a new key `a\b` lands on
the same xl.meta as `a/b` and the two keys overwrite each other.
Reject '\' only where a key enters the namespace (check_new_object_args:
PutObject, cross-key CopyObject destinations, CreateMultipartUpload) and
only on Windows. Reads, deletes, in-flight multipart operations, in-place
copies, data movement and internal meta buckets keep accepting it, so keys
already stored with '\' stay readable, deletable and movable.
Pending owner ruling: whether new '\' keys should be rejected at all, and
whether reads/deletes should keep accepting them.
* ci(windows): quote filtered test runs and drop duplicate path
tempfile's Windows `persist` passes the raw paths to MoveFileExW and
SetFileAttributesW, bypassing std's extended-length handling. Once
`<object>/<rollback>/xl.meta.bkp` reached 260 UTF-16 units every delete
of that key failed with 500, while PUT and GET kept working.
Add `rustfs_utils::extended_length_path`, which normalizes a path with
GetFullPathNameW and prefixes it with `\\?\` (or `\\?\UNC\`), and use it
for the staging directory and destination handed to tempfile. Cover the
single and batch delete paths with a deep-key regression test and run the
helper's Windows assertions in the Windows filesystem workflow.
Fixes#8545
* fix(ci): prepare pinned Compose for Connect acceptance
Install the verified Compose CLI through one local action shared by all 11 connected acceptance entry points. Publish only checksum-verified binaries and keep daemon, provenance and timeout gates unchanged.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ci): use daemon-visible Connect temporary inputs
Match the existing official Connect acceptance TMPDIR contract in all 11 real E2E invocations so daemon bind mounts can read generated inputs. Preserve key modes, read-only mounts and cleanup checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
The shared connected preflight uses uuid.uuid7, which is unavailable in older runner Python environments. Select Python 3.14 with the existing pinned setup action in all eleven acceptance callers.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(build): pin gnu glibc baseline to 2.31 and report compile target in version metadata
gnu release binaries were built natively on Ubuntu 24.04, embedding
GLIBC_2.38/2.39 symbol references while the RPM/DEB metadata promised
glibc >= 2.31, so Rocky/RHEL 9 (glibc 2.34) could not run them (#8062).
The two linux-gnu legs now build through cargo-zigbuild pinned to the
2.31 baseline (same-arch; zig acts as the linker), with post-build
guards asserting both the embedded version metadata and the highest
referenced GLIBC symbol version, keeping the package declarations
honest.
Version metadata itself now comes from cargo's TARGET cfg instead of
shadow-rs' host stamping, so builds whose host differs from their
target (cross compilation, musl targets on a glibc-hosted toolchain)
report the platform they were compiled for (#8048 follow-up).
The cross toolchain (zig 0.14.x, cargo-zigbuild 0.23.4) is pinned in
the setup action now that it is load-bearing for releases, the build
job gets an explicit matrix-independent name so the connect-*
acceptance workflows keep matching it when matrix fields change, and
the preview-release workflow contract is updated for the TARGET_ARG
build commands.
Fixes#8062
Follow-up on #8048
* fix(build): enforce GNU ABI floor and canonical target metadata
Pin Zig 0.17.0 and cargo-zigbuild, isolate cross-toolchain caches, and
build GNU releases against the declared glibc 2.31 floor. Validate both
release ELF binaries using their loader version requirements.
Generate target metadata once through shadow-rs and check the actual
release version output without adding a parallel metadata facade.
Refs: rustfs/backlog#2824
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(build): reuse GNU release profile across binaries
Switching the CLI back to the default strip profile restarted 419
compilation units after the server build. Keep the profiling build
profile for both binaries, then strip only the CLI artifact.
Retain serial links, server profiling symbols, and post-strip ABI gates.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(build): limit release artifacts to the rustfs server
Build only rustfs in the release workflow and remove the CLI build,
strip, and packaging paths. Preserve GNU profiling symbols and catalog
sidecars, and align strict acceptance archive readers with the new set.
Exercise all six build targets, Linux package checks, and all four
strict acceptance extractors using their actual workflow scripts.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(deps): patch profiling dependency for RUSTSEC-2026-0344
Vendor the published Pyroscope package with an object 0.40 dependency and
lock ruzstd 0.9.1. Preserve runtime sources and profiling features, add
bounded decoder regressions, and run the patched package in audit CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>
* fix(deps): correct vendored pyroscope documentation typos
Correct copied comments and changelog spelling for the repository typo
check, and describe these comment-only changes in the vendor provenance.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>
* fix(ecstore): refresh compact transition proof per test case
Install the short-lived homogeneous fleet proof immediately before each
kill-point transaction. Earlier cancellation and recovery work must not
age the next case's writer capability fixture.
Report an early transition result instead of waiting for an unreachable
kill point. Preserve cancellation, remote-owner and payload assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
Signed-off-by: Hauser <housemecn@gmail.com>
---------
Signed-off-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Merge main into gateway/integration every weekday with a --no-ff merge
commit; abort a conflicting merge and open a sync/<source>-<date> pull
request listing the conflicting files instead. Run ci.yml on pull requests
that target gateway/integration so the branch ruleset's required checks can
report, and document both in the CI gate matrix.
Refs rustfs/backlog#2736, rustfs/backlog#2734
timeout-minutes requires a numeric value, but inputs.pool_timeout_minutes
is a string, so '${{ inputs.pool_timeout_minutes || 240 }}' evaluated to
the string '240' and the runner rejected the template: 'An error occurred
when attempting to determine the step timeout ... Unexpected value 240'
(chain run 36505625402 job at 02:25:52, standalone run 36393453114 at
07:46:37). Both pool runs since #8155 merged have therefore executed
without the intended step budget. Wrap the expression in fromJSON and
update the policy test that pinned the broken literal.
* fix(ci): include pagination regression in full E2E selection
* fix(deps): replace yanked yoke-derive release
* fix(scanner): expose pause backlog replica diagnostics (#8258)
* fix(scanner): expose pause backlog replica diagnostics
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(connect): stabilize runtime profile lease cancellation
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(connect): tolerate delayed schedule startup in CI
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): retry quota reads during usage warmup
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): avoid meta-bucket incarnation self-deadlock
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(test): use persisted incarnation in heal fixture
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(usage): reconcile stale counters after lifecycle expiration (#8108)
* fix(usage): reconcile stale counters after lifecycle expiration
* fix(usage): account lifecycle expiry during continuous writes
* test(usage): run lifecycle usage scenarios on one scanner store
* test(usage): use a Windows-representable pre-mutation offset
* fix(usage): harden expiry accounting recovery and quota checks
Borrow expiry receipt bucket names and avoid allocating a map key on cache hits. Cover cancelled receipts, durable snapshot recovery, and legacy quota admission after scanner confirmation. Use representable timestamp offsets in the quota regression.
Refs rustfs/backlog#2689
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(usage): recover stale persisted counts through the scanner
Seed incorrect complete usage for empty and retained-object buckets, then run the real scanner and publication consumer without further object mutations. Verify durable and admin usage over two cycles instead of writing a corrected snapshot in the test.
Refs rustfs/backlog#2689
Refs rustfs/backlog#2691
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): bound delimiter pagination fixture concurrency
The 120-second smoke timeout expired after 1018 of 1200 serial fixture PUTs, before LIST ran. Prepare the same objects with at most eight concurrent requests and await every PUT. Retain the timeout and strengthen exact prefix, KeyCount, empty Contents, and continuation-token assertions with phase diagnostics.
Refs rustfs/backlog#2689
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(upgrade): establish a persisted previous-release baseline
Seed the pinned previous-release cluster and restart it once with its data intact before replacing any node. Require every old writer to pass the strict readiness probe and preserve the seed through both mixed phases and the final current cluster. Keep InternalError fail-fast behavior and all existing compatibility deadlines and assertions.
Refs rustfs/backlog#2689
Refs rustfs/backlog#2384
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): bound cancellation metadata persistence waits
Use the system-bucket incarnation boundary now supplied by main PR #8268. Bound the three cancellation waits that previously hung during pool.bin persistence, retaining their remote-generation, target-cohort, and durable-state assertions.
Refs rustfs/backlog#2697
Refs rustfs/backlog#2689
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Chris <anzhengchao@gmail.com>
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(e2e): retain startup and shutdown failure diagnostics
* fix(test): supply CPU workload for sampler regression
* fix(ci): locate security chain scripts in the workspace
* fix(usage): recover historical counters with generation fencing (#8273)
* fix(usage): recover historical counters during continued writes
Use newer converged scanner snapshots to reconcile stale absolute usage
baselines while preserving concurrent mutation and expiry receipt fences.
Cover durable publication, admin and quota reads, and legacy generations.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(usage): fence snapshots and move preserved cache entries
Apply the cached scanner generation floor before every reconciliation path
and retain it even when an older snapshot happens to match core counts.
Move preserved usage entries instead of cloning their histogram maps under
the cache lock, retaining expiry receipt identity and cancellation fences.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): verify checkpoint takeover and repair dispatch (#8275)
* test(scanner): cover checkpoint handoff and repair dispatch
Drive runtime budget expiry, partial-cycle persistence, leadership claims,
and stale checkpoint rejection between real disk-backed fixture scans.
Verify that a metadata repair beyond the first bounded prefix is saved in
the scanner ledger and dispatched by the MRF consumer.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* ci: isolate scanner fixtures and refresh full e2e membership
Reserve nextest capacity for the real-disk scanner publication and MRF
admission fixtures. Bind both platform membership checks to the reviewed
pagination deadline test added on main.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(scanner): consolidate checkpoint fixture lifecycle
Keep one durable control store across timeout and leadership transitions,
and inject generation advancement into the shared checkpoint scenario.
Check the actual saved metadata path so late-write rejection also proves
that existing checkpoint bytes remain intact.
Centralize MRF fixture isolation and reuse nextest process isolation when
the startup environment already satisfies the test contract.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(obs): distinguish allocator counters from live memory (#8274)
* fix(obs): distinguish allocator counters from live memory
Preserve count/counter semantics and mark requested-byte attribution unavailable when live statistics or sampling are missing. Document sustained multipart memory diagnosis.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(obs): parse allocator statistics from one node
Resolve each statistic before interpreting its shape, avoiding unsupported-field tree scans and mixing data across wrapper scopes. Preserve unavailable-statistics policy and add precedence regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): isolate late parity recovery from metadata hedges
Use the existing object-scoped hedge timer barrier in exact-count recovery fixtures. Preserve payload and total-read assertions and verify that the omitted parity disk is read only during late refresh.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(usage): combine identical snapshot retention branches
* fix(test): await HTTP sender readiness in Top RPC fixture
* [release/1.0.1] Gate multipart copy through write admission (#8284)
Gate multipart copy through write admission
Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* Gate multipart copy through write admission (#8283)
Make UploadPartCopy acquire the shared foreground write admission permit before lifecycle locks or source readers so server-side multipart copy cannot bypass the same backpressure used by UploadPart. Document the shared queue semantics and add focused coverage for saturation, cancellation, lock ordering, and disabled admission.
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix: add UploadPart OOM validation guardrails (#8287)
* docs(release): validate candidates on release branch
* fix(scanner): validate checkpoints against global cycle fence (#8278)
## Related Issues
Related to rustfs/backlog#2701.
## Summary of Changes
Route scanner checkpoint cycle and leader validation through the global store while retaining the owning set for cache persistence, CAS revisions and publication admission.
## Verification
Two independent final-diff source reviews found no issues across correctness, concurrency and durability, test coverage, compatibility, performance and simplicity on head `8b8fe51d092090b053f552ae283960e2e306be33`. Root approval `5373624714` is bound to that exact head. Regression tests cover real two-pool routing, stale fences, post-save rejection and CAS conflicts; their reported local execution belongs to the PR author, not this merge operation. Current required CI remains pending, and this authorized admin squash does not establish CI or runtime acceptance.
## Impact
Restores checkpoint progress when global cycle and leader state differ from a set-scoped view. No format, retry, timeout, assertion or scanner-policy changes are introduced by this diff. The three prior main scanner failures remain unproved repaired.
## Additional Notes
Full validation must run on the resulting exact main revision. Reverting this patch restores the earlier set-scoped fence lookup and its checkpoint rejection behavior.
* fix(ci): restore E2E membership and pagination timeouts (#8281)
* ci: locate the auto-testing checkout for lanes that run evidence from a subdirectory (#8279)
## Related Issues
Follow-up to #8229.
## Summary of Changes
Locate the private auto-testing checkout from the lane root or the workspace root so the nested security checkout can record functional-chain evidence.
## Verification
The exact PR head b66129ab9f passed one mechanical correctness and simplicity review, nine real-Git layout and provenance checks, and sixteen existing evidence/envelope tests. The baseline sibling layout failed with git exit 128; the corrected layout succeeded while revision mismatches and missing checkouts stayed rejected. Current PR checks are completed with successful or skipped conclusions, including the aggregate.
## Impact
Both lane and private-script revision checks remain intact. No time limits, assertions, production behavior, or evidence validation requirements change. The synthetic layout checks do not execute the actual scheduled security suite; integrated main CI and release acceptance remain separate gates.
## Additional Notes
Approved review 5374559393 is bound to the exact head above. Reverting the single-file change restores the previous checkout lookup.
* fix: add UploadPart OOM validation guardrails
Add a Docker validation harness for backlog#2704 so the ordinary UploadPart
low-concurrency memory workload can be reproduced with comparable case metadata,
process/cgroup sampling, TLS and metrics toggles, cache-env controls, and write
reclaim/direct-write experiments.
Warn when operators set the unrecognized RUSTFS_OBJECT_CACHE_* variables that
appeared in the reporter compose file. The variables are reported but remain
ignored, so startup does not silently change object data cache behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: AL <allan.bednarowski@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): pass fence store to checkpoint fixture
* fix(s3): queue bucket operations and restore strict Clippy checks (#8290)
* fix(s3): queue concurrent bucket creation and deletion
Keep eight active bucket transactions and bound admission waiting to 128 requests and 30 seconds. Preserve detached transaction ownership and return Retry-After with overload responses.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): restore strict Clippy compatibility on Rust 1.99
Use try_update without changing atomic ordering or overflow behavior. Keep
recursive storage futures boxed once at each frame and remove the redundant
async-recursion macro, including its non-recursive SQL planner use. Remove
needless closure borrows and orphaned dependency entries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore: refresh dependencies and atomic update APIs
Update workspace dependencies and the lockfile. Replace deprecated
atomic fetch_update aliases with try_update while preserving closures
and memory ordering.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* improve
* fix(scanner): diagnose and verify pause backlog recovery (#8293)
* fix(scanner): diagnose and verify pause backlog recovery
Expose the retained replica snapshot and claimed membership in abnormal
admin status responses. Keep diagnostics off metrics updates and verify
single-pool recovery and conflicting-proof preservation across 24 sets.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(scanner): move replica snapshots into diagnostics
Consume the terminal admin read snapshot in a single state match and move
membership, revision, and error buffers into the response. Verify buffer
handoff and the unchanged JSON contract without altering ledger authority.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(heal): wait for held legacy responsibility in replay test
* ci: remove Docker Hub description sync
* fix: emit NextPartNumberMarker only when ListParts is truncated
ListPartsInfo.next_part_number_marker was a non-optional usize that
defaulted to 0 and was only assigned when the response was truncated.
The S3 serializer then emitted it unconditionally as Some(0), causing
AWS SDK paginators to loop infinitely on part_number_marker=0 instead
of terminating.
Change the field to Option<usize> (None by default) and set it only
inside the is_truncated branch. The S3 output layer now uses
.and_then() so NextPartNumberMarker is absent when IsTruncated=false,
matching AWS S3 behavior.
Fixes#8208
(cherry picked from commit 44de803a38)
* fix(s3): honor sparse ListParts markers and verify termination
Resume part listings at the first part above the numeric marker, even
when that marker is absent. Use binary search over the sorted part
numbers and retain the existing exact-tail empty-slice path.
Add storage, XML, and real AWS SDK paginator regressions for empty and
terminal pages, sparse markers, and multipart completion integrity.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 673031eea1)
* fix(storage): publish delete rollback backups atomically
Stage rollback metadata outside the rollback directory and publish it only after the full write succeeds. A short write must not leave a backup that quorum rollback can rename over acknowledged version history.
Add an isolated real short-write regression and register the backported ListParts SDK test in the smoke and Linux full inventories.
* test(e2e): register paginator regression in Darwin inventory
* fix(release): install yq before Helm template checks
* chore(release): align installation references for 1.0.1
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: Peder Bergan <pederbe@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: AL <allan.bednarowski@gmail.com>
Co-authored-by: hector <42570491+majinghe@users.noreply.github.com>
Co-authored-by: Chapman <touch65536@gmail.com>
* fix(test): establish a writable previous-release upgrade baseline
* fix(ci): reserve capacity for durable admin fixtures
* fix(ci): group durable IAM state fixtures by resource needs
* ci: run E2E doctests with the E2E dependency graph
* fix(ci): separate fixture startup from transport deadlines
* fix(ci): bound pagination after seeding and revisit restored copies
* fix(ci): make filesystem fixture timing deterministic
* fix(ecstore): avoid metadata lock reentry during internal mutations
* test: align recovery fixtures with durable ownership contracts
Include helm/rustfs/README.md in the uploaded helm-package artifact so
publish-helm-package updates the root README.md in rustfs/helm.
The build-helm-package job copies helm/README.md into helm/rustfs/
before running helm package, but upload-artifact previously uploaded
only helm/rustfs/*.tgz. Because both files share the helm/rustfs/
parent directory, actions/upload-artifact places both the chart
archive and README.md at the artifact root, and download-artifact
extracts README.md into the root of rustfs/helm alongside the tarball.
* fix(ci): reduce duplicate work and preserve reliable test failures
* fix(ci): retain protocol evidence and repair stale test fixtures
* test(connect): honor parent deadline during API fixture readiness
* fix(ci): reserve IO capacity for state writer proofs
* test(connect): align RPC fixtures with service capture contracts
* test(connect): cover pinned service capture failures
The security workflow checks the repository out into rustfs-repo/ (#7212)
but its chain evidence steps still invoke
scripts/functional_chain_evidence.py relative to the workspace root, which
on the persistent shared runner resolves to a stale checkout left by another
job. The evidence gate then compares that checkout's HEAD against the chain
workflow_sha and rejects the lane before any case runs
("lane checkout differs from chain workflow source").
Run the evidence script from the lane's own checkout so ROOT resolves to
rustfs-repo/, whose HEAD is exactly the chain-pinned workflow_sha.
* fix(ci): restore mainline and scheduled test reliability
* fix(ci): provide GitHub CLI for CPU acceptance
* ci: provide Docker for CPU service acceptance
* ci: provision Python and Docker for OIDC validation
* ci: restore hosted runners for Docker validation
* ci: use verified MinIO release packages for interop
* ci: preserve host ownership of MinIO fixtures
* fix(ci): correct diagnostic limits and isolate startup checks
* test(connect): include object CLI failure details
* test(readiness): initialize unavailable drive diagnostics
Chris