Commit Graph

13 Commits

Author SHA1 Message Date
Hauser Zhi f2003af73d fix(ecstore): recover deletes on full disks (#9465)
* fix(ecstore): reserve emergency space for delete journals

Keep bounded physical metadata headroom for recoverable deletion on Unix without bypassing quorum or rollback.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* refactor(ecstore): recover delete metadata space on demand

Share a pinned reserve owner per physical disk and keep healthy deletion free of reserve filesystem probes. Retry only disposable staging and idempotent metadata primitives after actual ENOSPC, preserving transaction publication and quorum boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): retry concurrent full-disk deletes

Serialize ENOSPC waiters per physical disk and give each safe metadata primitive one bounded retry against shared released capacity.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ci): account for delete reserves in receipt fixture

Reject undersized tmpfs mounts after including the eight per-disk delete reserves and keep a self-test for the capacity boundary.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(heal): refresh resume admission fixture lease

Refresh the existing node admission sample immediately before starting resume GC so the cancellation test reaches the native-open ownership boundary.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): inject receipt ENOSPC on hardlink path

Keep the receipt fault at the atomic allocation boundary after the stage-two hardlink fast path landed, so reserve recovery remains covered.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-17 07:53:38 +00:00
Chris An 1cb4c0ebe2 ci: stamp the plan contract and rename the workspace lint-and-build key (#9095)
* ci: gate the plan on a contract marker

A base planner predating the NIGHTLY_JOBS split still emits the old required
set, and the strict verifier would reject it. Select now stamps plan_contract=2
and the workflow refuses any plan without the current marker, falling back to
the full matrix, so a stale base planner can neither authorize the new narrower
selection nor force a job the new workflow no longer schedules.

* ci: rename the workspace lint-and-build job key and evidence path

The job key test-and-lint named a job that runs no tests: it does Clippy, the
test-binary archive, doctests and the log-rule anchors, while the tests live in
workspace-tests. Anyone reading needs: [test-and-lint] would conclude tests had
already gated that dependency. The key becomes workspace-lint-and-build to
match its display name, and the shared evidence directory becomes
artifacts/workspace so the path stops asserting the same thing. The aggregate's
display name stays Test and Lint because that is the live ruleset check
context. ci-gates.md also stops attributing the ECStore test run to the wrong
job.
2026-10-31 17:41:02 +08:00
Chris An 191c95a469 ci: enforce affected validation and rehearse bounded shards (#9070)
* ci: enforce affected validation and rehearse bounded shards

* test(ci): keep regression output stable under forced color
2026-10-30 11:58:38 +08:00
Chris An 4fcbf60f10 ci: unify validation planning, artifacts and suite orchestration (#9059)
* ci: unify validation planning, artifacts and suite orchestration

* fix(ci): preserve JSON diagnostics for release checks

* fix(ci): keep manual attempts without plans visible

* fix(ci): distinguish nextest setup scripts from test results

* ci: reuse extracted archives and consolidate duplicate checks
2026-10-30 02:02:38 +08:00
Hauser e0444b519d feat(ecstore): add sampled batch CPU diagnostics (#9019)
* feat(ecstore): add sampled batch CPU diagnostics

Keep timing and request correlation in a separate default-off feature. Preserve the existing positioned-read backend, byte and error contracts.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): route batch diagnostics through storage facade

Keep the diagnostic implementation private to its owner and expose explicit feature-gated helpers through the existing storage boundary.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): retain diagnostic IDs across GET producer tasks

Capture request context before spawning the legacy producer. Keep the existing producer, cancellation and read-lock lifetime intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): avoid unused batch read index

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(obs): isolate local batch diagnostic JSON events

Omit span metadata and inferred parent IDs for diagnostic root events. Verify per-shard index order and document existing production metric timing.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): accelerate receipt fixture in CI

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(heal): allocate stack for bucket recovery fixture

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(s3): allocate stack for conditional GET fixture

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-29 10:32:06 +08:00
Chris 9749447825 test(connect): retain CPU profile import bundle (#7820) 2026-09-14 10:54:25 +08:00
Chris 874d0d7b63 test(connect): retain CPU profile verification key (#7819) 2026-09-14 10:50:42 +08:00
Chris d2e2aa7d0b test(connect): verify profile cpu release artifacts (#7818) 2026-09-14 10:48:17 +08:00
Chris b391ec4f0e ci: retain top disk acceptance archive (#7812) 2026-09-14 10:29:28 +08:00
Chris a0bbcba2b9 test(connect): require native top.disk runner (#7809) 2026-09-14 10:24:28 +08:00
Chris dfcee45b98 test(connect): verify top.disk release artifacts (#7808) 2026-09-14 10:21:37 +08:00
Zhengchao An b7805caa58 ci: stop archiving every dependency's unpacked source in each cache (#5566)
The four consolidated ci keys plus the build keys still do not fit the
repository's fixed 10GB Actions cache quota, so LRU keeps evicting them:
measured demand is ci-dev 2331MB + ci-feat-proto 2310MB + ci-feat-rio 1951MB +
ci-uring 1317MB + two build legs at ~1429MB + cargo-deny 844MB, and main pushes
add two more build legs. That is roughly 14.4GB against 10.24GB. The symptom is
misleading: Cache Warm reports every job successful and the restores log "full
match: true", yet ci-feat-rio and ci-uring disappear from the cache list between
runs.

cache-all-crates was the wrong default for this repository. With it set to true,
rust-cache's cleanup returns before pruning ~/.cargo/registry/src and its config
archives the whole registry, so every cache carried the unpacked source tree of
every dependency — not, as the name suggests, just a few extra crates.

Setting it to false is rust-cache's own default and loses no coverage: the
package set comes from `cargo metadata --all-features`, a strict superset of any
single lane's feature closure; -sys crates are explicitly exempt from pruning,
since their source timestamps would otherwise trigger rebuilds; and everything
pruned is re-unpacked from the .crate files still in registry/cache, whose
mtimes crates.io normalises, so cargo fingerprints stay valid.

Applied to the setup composite and to audit.yml's own rust-cache. Cache Warm
now also reports the sizes of registry/src, registry/cache, registry/index,
~/.cargo/git and target/ to the step summary, immediately before rust-cache's
post step archives them, so the size of the effect is measured rather than
assumed.

The new sizes only appear once the cache key next rotates, since rust-cache
skips the save entirely on an exact key hit. Deliberately not forcing that by
bumping prefix-key: it would invalidate every family at once and produce a
repository-wide cold build.

Refs: rustfs/backlog#1598, rustfs/backlog#1600
2026-08-01 18:57:23 +08:00
Zhengchao An 3f4f31129e ci: narrow the io_uring lane and make the sampler attributable (#5540)
Two independent changes to the Test and Lint area.

The io_uring lane compiled 7 integration binaries to run none of their tests.
Job log 91309868055 shows the lib target reporting "18 passed; 3453 filtered
out" while every binary under crates/ecstore/tests/ reported "running 0 tests".
Adding --lib drops them from the build without changing the selected set.

The `uring_` filter itself must not be touched. libtest matches on substring, so
it also selects names containing `during_` — 6 of the 18 selected tests are such
incidental matches, and narrowing the filter to `io_uring` would silently drop
them. scripts/check_uring_lane_lib_only.sh asserts the precondition --lib
depends on: no test function whose name contains `uring_` may live under
crates/ecstore/tests/. A count floor would not do, because the dangerous case —
someone adding a matching test there — leaves the lib count unchanged and CI
green.

The resource sampler was measuring the wrong machine. The runners are ARC pods,
so /proc/loadavg, /proc/pressure/*, `free` and `df` are node-level and include
every other runner pod on the same Kubernetes node: one sample reported loadavg
6.67 with 832 threads node-wide while `ps` inside the pod showed about 10
processes. Judging a CARGO_BUILD_JOBS change on those numbers cannot work.

The sampler moves to scripts/ci/resource_sampler.sh and now records
/sys/fs/cgroup cpu.max, memory.max, memory.peak and the cpu/io/memory pressure
files, which are this pod's own. The node-level readings stay — co-tenancy is a
real cause of stalls, and a 9m57s plain `git checkout` was traced to it — but
are labelled NODE-LEVEL so nobody reads them as this job's load. Phase markers
are written on start, and clippy is now sampled too: it is the natural control
arm for a CARGO_BUILD_JOBS experiment, since --all-targets is check-only for
workspace members and never links the ~100 test binaries the limit throttles.

No numbers are changed in this commit. CARGO_BUILD_JOBS stays at 2 until there
is attributable data to change it on.

Refs: rustfs/backlog#1598, rustfs/backlog#1601
2026-08-01 11:32:00 +08:00