Scheduled workflow failures previously went unnoticed (15 consecutive red
weekly s3-tests sweeps, silent perf nightly failures). Add a reusable
composite action that opens a tracking issue titled
"[scheduled-failure] <workflow name>" — or appends a comment to the
existing open one (dedupe key = workflow name) — with the run URL, run
attempt, and failed job names, labeled "infrastructure".
Wire it into the scheduled paths of e2e-s3tests, mint, fuzz (nightly) and
performance-ab via a final alert-on-failure job gated by
"always() && github.event_name == 'schedule' &&
contains(needs.*.result, 'failure')", with issues:write scoped to that
job only. e2e-s3tests and mint previously had no permissions key; they now
get workflow-level contents:read, reducing every other job's token.
Add a dispatch-only drill workflow that forces a job failure and runs the
action through the exact consumer wiring, for end-to-end verification.
The ci-7 e2e nightly does not exist yet; it is recorded as a pending
consumer in the action README.
Refs: rustfs/backlog#1149 (ci-8), rustfs/backlog#1155
Both nightly runs of the warp A/B rig failed at server bring-up:
`endpoint http://127.0.0.1:9000/health did not become healthy` — exactly 60s
after start. The server binds its public listener only after startup converges
(erasure-format load + IAM); its own budget
(RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS) defaults to 120s, so the rig's 60s
health poll gave up before a slow cold start on the shared sm-standard-2 runner
finished. The rustfs.log lived under /tmp (not the uploaded target/hotpath-ab/),
so the root cause was invisible in the artifact.
Root-cause + diagnosability (scripts/run_hotpath_warp_ab.sh):
- Health poll is configurable via --health-timeout, default 180s (> the 120s
server startup budget). Fails fast if the local server process exits before
becoming healthy instead of polling out the full window.
- Server logs + a startup-env file now write under OUT_DIR/server-logs/ so they
ride along in the CI artifact. On a health failure the rig dumps the last 50
log lines to the job log.
Workflow (.github/workflows/performance-ab.yml):
- On every run, write gate.md (or, on a pre-gate failure, the failing phase's
server-log tails) into $GITHUB_STEP_SUMMARY; short artifact retention.
- Short measurement matrix (--duration/--rounds/--cooldown) so the expanded
matrix fits; timeout-minutes 90 -> 120 as a Phase-0 stopgap until perf-3
caches the baseline binary (the ~65min double build dominates).
- TODO(ci-8/perf-2) hook comment for the failure-alert composite action.
Workload cells (absorbed perf-4): add put-4kib/get-4kib (the #4221 fsync
regression size, ~-10% @4KiB, previously invisible) and get-10mib (historical
large-GET EOF size) to both the PR-labeled and nightly matrices — 6 workloads
x 2 phases x 2 drive-sync = 24 cells. A 1KiB cell is deferred to protect the
budget until perf-3.
Refs rustfs/backlog#1152 (perf-1, absorbed perf-4), rustfs/backlog#1155.
The mint multi-SDK compatibility pipeline had exactly one recorded run
(28730530597, 2026-07-05), which died at "Enable buildx": the self-hosted
sm-standard-4 pod it landed on had no /var/run/docker.sock, so
`docker buildx create` failed to connect to the Docker API before any test
ran. Same heterogeneous-fleet root cause ci-1 fixed for the weekly s3-tests
sweep.
- Pin the job to GitHub-hosted ubuntu-latest, which reliably ships a running
Docker daemon + buildx + python3. Header note records the diagnosis and the
dind-sm-standard-2 tradeoff.
- Pin MINT_IMAGE default to the linux/amd64 digest for minio/mint:edge
resolved 2026-07-10 (was the rolling :edge tag); workflow_dispatch can still
override. Header comment records the refresh command.
- Quote shellcheck-flagged vars and drop an unused loop var so actionlint
passes with zero findings.
- Left report-only (ci-3 adds baseline gating later, per adjudication G4) and
a TODO(ci-8) alerting hook on the scheduled job.
Refs: rustfs/backlog#1149 (ci-2), rustfs/backlog#1155
test(ci): add strict nextest ci profile with quarantine + flake policy
Formalize the existing ecstore-serial-flaky mechanism into a strict CI gate
(ci-10, absorbs infra-15; backlog#1149).
- .config/nextest.toml: add [profile.ci] with global retries=0 (never mask a
new race's first occurrence), fail-fast=false, and JUnit output at
target/nextest/ci/junit.xml. Add a quarantine section where flaky tests get
retries=2 under the ci profile only; each entry links one OPEN issue. First
members are the two backlog#937 ecstore groups
(concurrent_resend_same_part_commits_one_generation and
store::bucket::tests::bucket_delete_*), which keep their existing
ecstore-serial-flaky test-group serialization. Local default profile still
never retries.
- .github/workflows/ci.yml: run the main test step with --profile ci and upload
the JUnit report (if: always(), 3-day retention, run-number in name). The
migration-proof step stays on the default profile to avoid clobbering the ci
JUnit artifact (its tests are not quarantined).
- docs/testing/README.md: new skeleton (owned by backlog#1153 infra-11) holding
the flake policy: discover -> open issue within 24h -> quarantine with issue
link -> fix or delete within 30 days. AGENTS.md points to it.
Refs: rustfs/backlog#1149, rustfs/backlog#937, rustfs/backlog#1155
The weekly full ceph/s3-tests sweep (e2e-s3tests.yml, schedule path) has
failed every recorded run. The 2026-07-05 run (rustfs/rustfs #28727691591)
died at "Install Python tools" with `No module named pip`: the self-hosted
`sm-standard-4` label is served by heterogeneous runners and the scheduled
job landed on a minimal pod with neither `pip` (bare python3) nor
`docker.sock`, so no test ever executed.
Make the infrastructure assumptions explicit instead of trusting drifting
self-hosted runner state:
- Pin the job to GitHub-hosted `ubuntu-latest`, which reliably ships Docker,
docker compose, python3 and pip.
- Add an explicit actions/setup-python step before any pip usage so the
workflow stays correct across runner-image drift.
- Document the fix and rationale in the workflow header comment.
Also quote the shell variables flagged by shellcheck (SC2086) in the
Docker build/run steps and drop the unused loop variable (SC2034) so
actionlint passes with zero findings on the changed file.
The PR gate (ci.yml s3-implemented-tests) is unaffected: it avoids Docker
via DEPLOY_MODE=binary and defers pip setup to run.sh's self-bootstrap.
Refs: rustfs/backlog#1149 (ci-1), rustfs/backlog#1155
Re-enable the Fuzz workflow (disabled_manually) with two changes that
address the congestion that likely caused it to be disabled:
- Narrow the PR trigger paths to fuzz/**, scripts/fuzz/** and
fuzz.yml only. The broad crate paths (crates/ecstore|filemeta|utils)
queued a ~45min fuzz-build on nearly every PR; coverage of those
crates moves to the nightly schedule, which fuzzes against main.
- Expand the smoke matrix, nightly matrix, run.sh default set and the
fuzz-build staging/upload steps from 3 to all 5 buildable targets:
archive_extract, bucket_validation, local_metadata, path_containment,
policy_ingress. archive_extract and policy_ingress were previously
in no matrix.
The two *_storage_api.rs files are `mod` submodules of their parent
targets (bucket_validation, path_containment), not standalone bins, so
fuzz/Cargo.toml registers exactly 5 fuzz bins.
Smoke jobs run one target per parallel matrix job (-max_total_time=60),
so wall-clock stays per-target, well under the 15min budget for a
fuzz-only PR. A TODO(ci-8) hook marks where scheduled-failure alerting
will attach on the nightly job.
Refs rustfs/backlog#1149 (ci-9, absorbed sec-13), rustfs/backlog#1155
* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
feat(ecstore): add runtime-probed io_uring read backend, gray-off by default (rustfs/backlog#1104)
Wire the standalone rustfs-uring crate (https://github.com/rustfs/uring) into
LocalIoBackend as an opt-in read backend. When RUSTFS_IO_URING_READ_ENABLE is
set AND the per-disk io_uring probe succeeds, positioned reads go through the
cancel-safe UringDriver; otherwise — default, or on a restricted host, or on
any per-read driver error — reads fall back to StdBackend byte-for-byte, so the
default build is unchanged.
- Cargo.toml: rustfs-uring as a Linux-only git dependency (pinned rev). The
guard scripts/check_no_tokio_io_uring.sh allows an explicit io-uring
integration; only the tokio "io-uring" runtime feature is banned.
- UringBackend mirrors StdBackend::pread_bytes's resolution/access/bounds
preamble and only swaps the raw byte read; the other three trait methods
delegate to the inner StdBackend.
- Backend selection at LocalDisk construction via build_local_io_backend.
- Differential test uring_backend_reads_match_std: with the flag set, every
positioned read returns byte-identical data (driver-served when io_uring is
available, StdBackend fallback otherwise).
Verified: cargo check -p rustfs-ecstore on Linux; the differential test passes
under real io_uring (seccomp=unconfined). The runtime errno degradation latch,
per-disk probe cache, and productionization are tracked in
rustfs/backlog#1101/#1102.
Co-authored-by: heihutu <heihutu@gmail.com>
Add a default-on multi-role adversarial validation policy to the root
AGENTS.md (risk tiers, six reviewer roles, findings protocol, exit
criteria), a shared adversarial-validation skill with repo-specific
attack probes grounded in shipped bugs, seven audit fixes to the root
instruction files, and an actionlint gate for workflow changes.
- build.yml: update latest.json only for stable release tags
(alpha/beta/rc tags previously overwrote the stable pointer with
release_type "stable"); drop the placeholder .asc files; build the
Linux targets only on main pushes (tags/schedule/dispatch keep the
full matrix); remove the unreachable --build clause and a needless
full-history clone
- ci.yml: event-scoped concurrency so merges stop cancelling the
weekly scheduled run; drop the redundant skip-duplicate-actions
gate job; run clippy before tests; trim the unused toolchain from
the typos job
- ci-docs-only.yml (new): satisfy the required "Test and Lint" check
on docs-only PRs that ci.yml skips via paths-ignore
- audit.yml: drop cargo-audit (cargo-deny advisories covers the same
RustSec database); same event-scoped concurrency fix
- docker.yml: job-level short-circuit for per-merge dev builds; send
the Trivy SARIF to code scanning; unify the upload-artifact pin
- performance-ab.yml: add concurrency; only re-run on labeled events
when the added label is perf-ab
- stagger the Sunday crons (build 01:00, audit 03:00,
nix-flake-update 05:00, mint 06:00)
- delete performance.yml: disabled since 2025-07; idle-server
profiling, no benchmark baseline, stale ecstore package name
* ci(perf): warp A/B relative-budget gate for the hotpath series
performance.yml only uploads a samply profile and a cargo-bench artifact with
no pass/fail, so a 26x-class write-path regression like #4221 or the GET perf
churn would sail through CI and only surface in customer load tests. This adds
the missing gate — the acceptance surface every queued HP change (#922/#923/
#925/#927/#930/#932) needs before its default can flip.
- scripts/hotpath_warp_ab_gate.sh: applies the relative budget to the
baseline_compare.csv that run_object_batch_bench_enhanced.sh already emits
(it computes deltas but never gates). A metric regressing past --fail-pct
fails, past --warn-pct warns; reqps/throughput are higher-is-better, latency
lower-is-better. --allow-regression downgrades a FAIL to an exempted WARN so
a deliberate correctness cost (e.g. #4221) is recorded, not blocked
(rustfs/backlog#935 correction 1). Unit-checked across pass/warn/fail/exempt.
- scripts/run_hotpath_warp_ab.sh: single-host baseline-binary vs candidate-
binary A/B over {put-4mib, get-4mib, mixed-256k} x {drive-sync on, off},
reusing the enhanced bench as the warp driver and the single-node local-disk
lifecycle. Has --dry-run; warp is assumed pre-installed as elsewhere in
scripts/. Drive-sync on/off keeps a sync-semantics change from being masked
by nosync numbers.
- .github/workflows/performance-ab.yml: nightly on main (post-merge detection)
plus opt-in pre-merge via the `perf-ab` label; `perf-deliberate-tradeoff`
runs the gate with --allow-regression; posts the gate table as a PR comment
and uploads the run. A Linux runner answers "do the macOS conclusions hold".
The gate logic is unit-validated with synthetic compare CSVs; the orchestrator
and workflow are shellcheck- and --dry-run-validated. The first real warp
measurement belongs on the Linux runner (no warp/multi-disk rig locally).
Refs: rustfs/backlog#935 (HP-14 warp A/B gate, item 4), rustfs/backlog#725
(cooled A/B harness precedent), rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
* ci(perf): pin upload-artifact, add external-cluster A/B mode + runbook
- Pin actions/upload-artifact to the repo-standard full-length SHA (# v6);
the previous @v4 float tripped the workflow-pin guard.
- Add an external deployment mode to run_hotpath_warp_ab.sh so warp can target
an already-running cluster instead of a throwaway single-node server:
--endpoint selects the cluster and --deploy-hook runs between phases to swap
in the phase's binary and drive-sync config (context passed via
HOTPATH_AB_PHASE / HOTPATH_AB_BINARY / HOTPATH_AB_DRIVE_SYNC). This maps onto
the team's ansible harness (cargo zigbuild -> ansible rustfs-manage --tags
stop,config,binary-copy,start -> warp).
- Restructure the matrix loop to bring a deployment up once per (phase,
drive-sync) and run all three workloads against it, instead of restarting per
workload — fewer server starts / cluster redeploys. Baseline medians are read
from a deterministic path, dropping the bash-4-only associative array so the
script (and its --dry-run self-check) runs on macOS bash 3.2 too.
- Add docs/operations/hotpath-warp-ab-runbook.md tying local mode, the ansible
external mode, and the budget/exemption together.
Verification: shellcheck clean; --dry-run in both modes prints the expected
4 deployments x 3 workloads and passes exactly 6 compare CSVs to the gate;
check_workflow_pins.sh, check_doc_paths.sh, and make pre-commit all pass.
Refs: rustfs/backlog#935, rustfs/backlog#936
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Drop the [target.'cfg(target_os = "linux")'.dependencies] tokio
"io-uring" feature from 6 crates and the rustfs binary. Because
.cargo/config.toml enables --cfg tokio_unstable globally, this feature
switched every Linux build's file I/O onto tokio's io_uring runtime
backend. Restricted Linux environments (Docker default seccomp, gVisor,
proot, old kernels) reject io_uring_setup with EACCES/ENOSYS, which
tokio surfaced as PermissionDenied and RustFS reported as
DiskAccessDenied at startup.
Add scripts/check_no_tokio_io_uring.sh so the feature cannot silently
return: it fails on any tokio dependency line enabling "io-uring", while
still allowing a future application-level io-uring crate dependency.
Wire it into make pre-commit/pre-pr/dev-check and CI.
Tracking: rustfs/backlog#890 (parent rustfs/backlog#897)
Co-authored-by: heihutu <heihutu@gmail.com>
Follow-ups to the compatibility harness rework:
- Weekly full sweep now runs as a matrix over both topologies: single
node and the 4-node distributed cluster. Manual dispatch keeps the
test-mode input.
- New mint workflow (weekly + dispatch) runs MinIO Mint against RustFS:
functional suites of real client SDKs and tools (awscli, mc,
aws-sdk-*, minio-*, s3cmd, ...), catching client-specific signing and
streaming edge cases that boto3-only ceph/s3-tests cannot. Report-only
for test failures with a per-suite summary table; fails only when mint
produces no results.
- New scripts/s3-tests/api_coverage.py quantifies API surface coverage
by diffing the s3s S3 trait (at the Cargo.toml pinned revision)
against the methods overridden in impl S3 for FS, distinguishing
NotImplemented defaults from delegating defaults (e.g. post_object).
Current state: 76/101 operations covered (75 overridden, 1 delegated).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Reworks the S3 compatibility test harness for reproducibility and faster
feedback:
- Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the
449-test PR gate is reproducible; previously every run cloned upstream
master, letting test renames or assertion changes break CI silently.
- PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single
CI round reports every failure in parallel instead of stopping at the
first one serially.
- Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and
report_compat.py to diff junit results against the classification
lists (regressions, promotion candidates, unclassified tests).
- Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source
of truth; also fixes the broken config generation that left S3_PORT
empty), add a weekly scheduled full sweep that fails only on whitelist
regressions, and fix the multi-node topology to a real distributed
cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent
single-node stores behind a load balancer.
- Docs: rewrite stale .github/s3tests/README.md (marker-era strategy),
update scripts/s3-tests/README.md, fix dead build_testexpr.sh
reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt.
All 747 classified test names verified present at the pinned revision;
13 upstream tests are currently unclassified and will surface in the
first full-sweep report.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Agent-instruction and architecture docs had drifted from the code:
- CLAUDE.md: slim to commands + pointers; fix wrong claim that
`make pre-commit` is the full pre-PR gate (that is `make pre-pr`);
drop stale pre-#3929 file paths and merged bug narratives
- AGENTS.md: drop dead `rust-refactor-helper` skill rule and the
hand-maintained (already stale) scoped-AGENTS index; link
architecture docs from Sources of Truth
- .github/AGENTS.md: replace the outdated copied CI command matrix
with a pointer to ci.yml
- crates/AGENTS.md: merge duplicated Testing sections
- ARCHITECTURE.md: resolve the utils->config contradiction (edges are
removed), mark volatile counts as snapshots, fix a bad path
- docs/architecture: add README router; move one-shot plans/trackers
(rebalance-decommission phases, migration-progress ledger, PR
template) to docs/superpowers/plans with archive headers; fix stale
source paths in kept inventories (core/sets.rs, core/pools.rs,
store/mod.rs, startup_* split from #3671)
- docs/operations/tier-ilm-debugging.md: extracted tier debugging
playbook with corrected paths
- scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when
instruction/architecture docs reference nonexistent file paths
- .claude/skills: add tier-debug and arch-checks repo skills;
.gitignore now keeps .claude/skills and docs/operations committable
Verification: ./scripts/check_doc_paths.sh,
./scripts/check_architecture_migration_rules.sh (both pass)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Speeds up the CI pipeline without changing what is tested:
- Drop the global CARGO_BUILD_JOBS=2 and per-job --jobs 2 flags, which
halved compiler parallelism on the 4-core runners.
- Stop embedding hashFiles(Cargo.lock) in rust-cache shared keys. The
action already fingerprints lockfiles internally; putting the hash in
the key prefix meant any PR that touched Cargo.lock started from a
completely cold cache instead of reusing unchanged dependencies.
- Give the e2e-tests job the full setup action with dependency caching.
Its migration-proof step compiles the e2e_test crate (which pulls in
most of the workspace) and previously did so cold on every run with
no cache, on a 2-core runner.
- Set profile.dev debug = "line-tables-only": keeps usable backtraces
while cutting compile/link time, target-dir size (faster cache
save/restore), and debug-binary artifact upload/download.
- Split fmt + repo-script checks into a compile-free quick-checks job
on ubuntu-latest so contributors get feedback in ~1 minute instead
of after the full test run.
- Collapse the five per-filter migration-proof cargo test reruns into
one filtered nextest invocation; the same tests already run in the
full nextest pass.
- Remove the touch rustfs/build.rs + forced rebuild from the debug
binary jobs (version stamping is irrelevant for e2e binaries) and
the unreachable Windows cross-compile branch in build.yml.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Zhengchao An