docs: remove agent-generated planning docs, forbid committing them
Delete one-shot planning/progress artifacts that were checked into the tree:
the 14 superpowers plan/tracker docs under docs/superpowers/plans/, plus
issue-scoped implementation plans, optimization conclusions, and dated
benchmark-result snapshots under docs/ (issue-4003 ListObjectsV2 plans,
get-small-file conclusion, issue824/issue829 benchmark results, issue-713
>1GiB GET baseline summary and ops guide).
Codify the rule so they do not come back:
- .gitignore drops the docs/superpowers whitelist, so anything new under
docs/ stays ignored unless force-added.
- AGENTS.md gains an explicit 'do not commit planning-type documents' rule
scoping version control to the durable architecture/operations/testing sets.
- docs/architecture/README.md, overview.md, arch-checks SKILL.md, and
check_doc_paths.sh drop their references to the removed archive.
The ecstore/global-state migrations are done (backlog#815, #939, #1052 all
closed). Review of every migration-era test/gate measure found three things
actually retirable or broken — everything else is a live anti-regression
guard and stays.
Remove:
- scripts/check_metrics_migration_refs.sh — guards a migration that
finished: rustfs_metrics:: has zero hits, the metrics crate no longer
exists, and the script was never wired into CI or make (only reference
was one line in config-model-boundary-adr.md, also removed).
- crates/obs init_metrics_collectors — the "backward-compatible alias kept
during migration" the removed script was guarding. Zero callers; pure
delegate to init_metrics_runtime.
Archive (docs/superpowers/plans/, continuing the 2026-07 convention,
with the standard archived banner):
- startup-timeline.md, scheduler-baseline.md,
profiling-numa-capability-inventory.md,
kms-development-defaults-inventory.md — one-shot snapshots whose only
consumer is the already-archived migration-progress ledger (their
same-dir links there start resolving again after the move); zero script
pins; fed the closed backlog#660/#665 architecture-review ledger.
Fixed the one outbound link (startup-timeline -> readiness-matrix) that
the move would have broken — check_doc_paths.sh deliberately does not
scan plans/, so nothing else would have caught it.
Wire (found orphaned by the same review):
- scripts/check_extension_schema_boundaries.sh guards a live contract
crate but was never invoked anywhere. Add lint-fmt.mak target, include
in pre-commit/pre-pr/dev-check, add ci.yml Quick Checks step (job
already installs ripgrep), sync the CONTRIBUTING.md enumerated list,
and harden the script against a silently-passing rg probe when src/
is missing.
Keep (verified live, documented so the next cleanup pass does not repeat
this analysis):
- scripts/check_architecture_migration_rules.sh — added a header stating
it is a permanent boundary guard, not retirable migration scaffolding;
'migration' in the name is historical.
- check_migration_gate_count.sh + floor, delete-marker e2e proof, all
pinned docs, compat-cleanup-register sync, remaining inventories
(referenced by live docs).
Verification: all 7 guard scripts pass, actionlint clean,
cargo check --workspace (excl e2e) clean, cargo fmt --check clean.
Adversarially reviewed by two independent skeptic passes; their 7
findings (alias left behind, broken outbound link, missing banners,
wrong backlog attribution, CONTRIBUTING drift, rg exit-2 hole, missing
header rationale) are all folded in.
* ci(ilm): move first batch of 5 s3-tests lifecycle expiration cases into a gated behavior lane (backlog#1148 ilm-10)
The 20 lifecycle cases in excluded_tests.txt were labeled "vendor-specific"
but the real blocker was the absence of a Ceph lc_debug_interval equivalent.
RUSTFS_ILM_DEBUG_DAY_SECS (ilm-5) now provides that, so Days>=1 expiration
behavior is testable in seconds.
These cases assert that objects/versions/uploads are actually removed by the
background scanner and the stale-multipart cleanup loop. They cannot join the
default single-server s3-implemented-tests gate: it disables the scanner, and a
global RUSTFS_ILM_DEBUG_DAY_SECS also shrinks the x-amz-expiration header that
the already-passing test_lifecycle_expiration_header_* cases assert on. So this
adds an isolated lane instead of putting them in implemented_tests.txt.
First batch (5), each verified against the pinned upstream source and the
RustFS evaluator/scanner/stale-multipart execution paths:
- test_lifecycle_expiration (prefix Days=1/Days=5, scanner delete)
- test_lifecyclev2_expiration (same via ListObjectsV2)
- test_lifecycle_noncur_expiration (NoncurrentVersionExpiration)
- test_lifecycle_deletemarker_expiration (ExpiredObjectDeleteMarker cascade)
- test_lifecycle_multipart_expiration (AbortIncompleteMultipartUpload)
Dropped from the batch, kept excluded with reason:
- test_lifecycle_expiration_days0: RustFS accepts Expiration{Days:0} (returns
200; only Days<0 is rejected in crates/lifecycle/src/core.rs validate()),
while AWS/this test expect InvalidArgument. Real validation gap.
Changes:
- scripts/s3-tests/lifecycle_behavior_tests.txt: new exact-node-id run set.
- scripts/s3-tests/run.sh: honor an IMPLEMENTED_TESTS_FILE override so a lane
can point at an alternate whitelist.
- .github/workflows/ci.yml: new s3-lifecycle-behavior-tests PR-gate job that
starts rustfs with RUSTFS_ILM_DEBUG_DAY_SECS=10, scanner enabled (cycle 2s,
no start delay) and a 2s stale-multipart cleanup interval, running the new
list serially.
- scripts/s3-tests/report_compat.py: recognize the behavior list so the weekly
scope=all sweep (plain server) does not misclassify expected failures.
- scripts/s3-tests/excluded_tests.txt: remove the 5 enabled cases; re-annotate
the remaining 15 lifecycle exclusions with real reasons + batch-2 plan.
- docs/architecture/s3-compatibility-matrix.md: sync counts (implemented 451,
excluded 274, behavior 5) and document the lane.
Refs backlog#1148 (ilm-10), master plan backlog#1155.
* fix(lifecycle): reject zero-day expiration/noncurrent/abort rules (backlog#1148 ilm-10) (#4722)
* refactor(obs): make dial9 telemetry opt-in and actually record events
The dial9 Tokio-runtime profiler was disabled by default, yet every build
paid for it, and enabling it produced trace files with no events in them.
Recorded empty traces
---------------------
`build_traced_runtime` called `TracedRuntime::builder()...build(..)`, but dial9
only starts recording in `build_and_start*`. `build` still returns a live guard
whose `is_enabled()` reports true, and still creates and seals segment files —
they just contain a header and no events. It also skipped `with_trace_path`, so
the background worker driving the segment pipeline was never spawned.
Measured on the new smoke example: 310 bytes of bare segment header, against
5640 bytes for the same workload once recording actually starts.
Switch to `with_trace_path(..).build_and_start(..)`.
Cost was unconditional
----------------------
`--cfg tokio_unstable` was a global `[build] rustflags` entry and `rustfs-obs`
depended on `dial9-tokio-telemetry` unconditionally, so all builds depended on
Tokio's non-semver API. Worse, an environment `RUSTFLAGS` replaces (never
appends to) the config-file value, so any caller exporting their own RUSTFLAGS
silently dropped the flag — the long comment in build.yml was a scar from that.
dial9 is now an opt-in feature (`dial9`, plus `dial9-s3` and `dial9-taskdump`),
the global rustflag is gone, and `crates/obs/build.rs` fails the compile if the
feature is on without the flag. Telemetry builds go through `make build-profiling`.
Metrics that could not lie
--------------------------
`rustfs_dial9_{events_total,bytes_written_total,rotations_total,cpu_overhead_percent}`
were hard-coded to zero — a Counter pinned at 0 reads as "nothing happened".
Removed. `rustfs_dial9_enabled` was sourced from the environment, so it read 1
even when the traced runtime failed and the process fell back to a standard
runtime; it is replaced by `rustfs_dial9_supported` (compile-time),
`rustfs_dial9_configured` (intent) and `rustfs_dial9_active_sessions` (reality).
No `writer_healthy` gauge is exported: dial9's `RotatingWriter` can enter its
`Finished` state and stop writing, but exposes no way to observe that, so the
gauge could only ever be hard-coded to 1. Documented as a known gap instead.
Final events were lost
----------------------
The `TelemetryGuard` lived in a `static OnceLock`, which is never dropped, so
buffered events were never flushed at exit. `build_tokio_runtime` now returns
the guard and `run_process` drops it before any exit path.
Also
----
- `disk_usage_bytes` was a `read_dir` + per-file `stat` on the metrics
collection path. It is now sampled by a background task into an atomic.
- `SAMPLING_RATE`/`S3_BUCKET`/`S3_PREFIX` were parsed, warned about, and
discarded. S3 upload is now wired to dial9's `with_s3_uploader` behind
`dial9-s3`; `SAMPLING_RATE` has no upstream equivalent and is removed.
- Wire `with_task_dumps` (async backtraces of stalled tasks), configurable via
`RUSTFS_RUNTIME_DIAL9_TASK_DUMP_{ENABLED,IDLE_THRESHOLD_MS}`.
- Split `telemetry/dial9.rs` into `config`/`state`/`enabled`/`disabled`; the
stub keeps the public API identical so callers need no `#[cfg]`.
- Drop four print-only examples and the manual test bin that exercised the
removed `init_session` scaffolding.
Verified: cargo check/clippy/test across default, `dial9`, and `dial9-s3`;
build.rs correctly rejects `dial9` without `--cfg tokio_unstable`;
`make pre-commit` passes.
Co-Authored-By: heihutu <heihutu@gmail.com>
* docs(obs): document dial9 as an on-demand profiler
scripts/run.sh advertised a `SAMPLING_RATE` knob that was never passed to dial9,
and claimed "CPU overhead < 5% (with sampling rate 1.0)" and "lower values reduce
CPU overhead" on the strength of it. The knob is gone; the guidance built on it
had to go too.
Replace it with what is actually true: dial9 needs a `make build-profiling`
binary, its disk budget evicts oldest-first (so a high poll rate can overwrite
the incident you are chasing), and it cannot be toggled without a restart.
Add docs/operations/dial9-runtime-profiling.md covering the build variants, an
investigation walkthrough, the configuration table, how to read the three
supported/configured/active_sessions gauges against each other, and the upstream
gap that makes writer death only indirectly observable.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(obs): add a dial9 smoke example that proves events are recorded
The bug this guards against is invisible to every existing signal: with
`build` instead of `build_and_start`, dial9 creates the trace file, seals
segments, and reports `TelemetryGuard::is_enabled() == true` — it simply
records no events. Only the segment's byte count tells the two apart.
Measured on this workload: 5640 bytes when recording, 310 bytes (a bare
segment header) when not. The example asserts >= 2048 bytes, and was verified
to fail with the `build` call restored.
Also correct the comment on the `is_enabled` check in `finish_traced_runtime`.
It claimed to catch "recording silently off"; it does not. It only rejects the
inert guard a lenient config yields after a build failure. Recording is
guaranteed by `build_and_start`, not by that check.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(rustfs): accept Unsupported runtime telemetry capability
A binary built without the `dial9` feature now reports the runtime-telemetry
capability as `Unsupported` rather than `Disabled`. The distinction matters to
operators: `Disabled` implies the capability can be switched on by setting an
environment variable, which is not true here — telemetry needs a rebuild.
Widen the assertion and pin the new semantics: when `dial9::is_supported()` is
false, the state must be exactly `Unsupported`.
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix(obs): drop the dial9-s3 feature, its TLS stack is vulnerable
CI's Dependency Review and `cargo deny` both reject the branch: dial9's
`worker-s3` feature depends on aws-sdk-s3-transfer-manager 0.1.3, which pins
aws-smithy-http-client onto hyper-rustls 0.24 and rustls-webpki 0.101.7. That
webpki carries RUSTSEC-2026-0098, -0099 and -0104.
0.1.3 is the latest release of the transfer manager, and 1.2.0 the latest of the
smithy client, so there is nothing to upgrade to. Cargo's feature unification can
add features but cannot drop a transitive dependency, so it cannot be worked
around from here either — the rest of the workspace already resolves to the safe
rustls-webpki 0.103 / hyper-rustls 0.27.
Remove the `dial9-s3` feature and the `with_s3_uploader` wiring. The two S3
environment variables stay parsed and warned about, now naming the real reason
rather than a missing build feature. Trace segments are collected from the output
directory instead. Tracked as D9-14 in rustfs/backlog#1157.
With this, Cargo.lock is byte-identical to main: the PR no longer touches the
dependency graph at all.
Also correct the `dial9-taskdump` documentation. It claimed the feature "compiles
to a no-op elsewhere"; in fact `tokio/taskdump` raises a `compile_error!` on any
target other than linux/{aarch64,x86,x86_64}. Verified by trying to build it on
macOS, which is how the claim was found to be wrong.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
refactor(concurrency): remove zero-caller facade modules and fix no-default-features build (backlog#1025)
The audit in rustfs/backlog#1010 (consistent with #805) established that most of crates/concurrency was a decorative facade with zero production callers; the real runtime concurrency control lives in rustfs/src/storage/*. This deletes the dead facades and keeps only what the workspace actually consumes.
Deleted (zero callers verified by workspace-wide grep):
- manager.rs: ConcurrencyManager, lifecycle start/stop, misleading 'started' lifecycle logs
- config.rs: ConcurrencyConfig, ConcurrencyFeatures, from_env
- timeout.rs: TimeoutManager, TimeoutGuard, TimeoutManagerPolicy
- lock.rs: LockManager, LockScopeGuard, OptimizedLockGuard
- scheduler.rs: SchedulerManager, SchedulerPolicy, IoStrategy
- deadlock.rs facade: DeadlockManager, RequestTracker
- backpressure.rs facade: BackpressureManager, BackpressurePipe
- the prelude module, unused io-core re-exports, and all feature flags
Kept (real callers in ecstore/heal/rustfs):
- workload.rs admission contract types (unchanged)
- workers.rs Workers pool (unchanged, retained per #4498)
- GetObjectQueueSnapshot (moved from manager.rs to new queue.rs)
- PipeBackpressurePolicy (used by rustfs/src/storage/backpressure.rs)
- DeadlockMonitorPolicy (used by rustfs/src/storage/deadlock_detector.rs)
- OperationProgress re-export (used by rustfs/src/storage/timeout_wrapper.rs)
Removing the feature flags fixes the previously broken cargo check -p rustfs-concurrency --no-default-features (E0432). Docs and the logging guardrail file list are updated to match.
Ref: rustfs/backlog#1025
test(interop): real-MinIO read + migration parity, Phase 1/2 (backlog#580)
Capture authentic on-disk fixtures from MinIO RELEASE.2025-07-23 (a bucket with
versioning, object-lock, lifecycle, tagging, quota, a public policy, SSE-S3
encryption, a webhook notification target, and a replication rule, plus inline /
versioned / multipart objects and a delete marker) and prove RustFS reads and
migrates them losslessly:
- filemeta parses_real_minio_object_xlmeta: small inline, two-object-version +
delete marker, and multipart object xl.meta parse to the expected FileInfo.
- ecstore parses_real_minio_bucket_metadata_blob_without_loss: the MinIO
.metadata.bin msgpack decodes via the PascalCase field names and
parse_all_configs loads all ten config types present (policy, lifecycle incl.
<ExpiryUpdatedAt>, object-lock, versioning, tagging, quota, notification,
encryption/SSE-S3, replication incl. DeleteMarkerReplication /
ExistingObjectReplication) without loss.
- ecstore reads_minio_inline_bucket_metadata_via_bitrot: MinIO inlines an object
body as [HighwayHash256 32B][body]; RustFS's BitrotReader with HighwayHash256S
verifies and yields the exact blob (the "inline_data 前缀不同" is that prefix).
- ecstore migrates_real_minio_bucket_metadata_end_to_end: on a throwaway 4-drive
local ECStore, a real MinIO .metadata.bin seeded under .minio.sys is migrated
into .rustfs.sys byte-identically for every config, exercising the Phase 2
source adapter (MIGRATING_META_BUCKET = ".minio.sys") through the object layer.
All four run as ordinary crate tests (nextest CI). Phase 4 (MinIO re-reading a
RustFS drive) is documented as out of scope for one-way migration.
Refs rustfs/backlog#580
Add two durable architecture references grounded in current code:
- minio-rustfs-router-compatibility.md: MinIO cmd/api-router.go (S3
object/bucket) and cmd/admin-router.go (admin /v3, /v4) vs RustFS
implementation status, with per-row landing points and a gaps-only
checklist of still-missing admin endpoints (profiling, healthinfo,
LDAP IDP config, replication diff, MRF, batch, locks, speedtest,
log stream, top, trace).
- minio-file-format-compat.md: xl.meta (meta_ver 3 write, <=3 read,
XL2 magic, rs-vandermonde, HighwayHash256 bitrot, inline data) and
.metadata.bin bucket-metadata interop matrix for the backlog#580
items, plus old-RustFS -> new migration and a phased plan.
Cross-links s3-compatibility-matrix.md and admin-route-action-snapshot.md
and indexes both docs in docs/architecture/README.md.
Refs rustfs/backlog#596rustfs/backlog#603rustfs/backlog#580
Agent-instruction and architecture docs had drifted from the code:
- CLAUDE.md: slim to commands + pointers; fix wrong claim that
`make pre-commit` is the full pre-PR gate (that is `make pre-pr`);
drop stale pre-#3929 file paths and merged bug narratives
- AGENTS.md: drop dead `rust-refactor-helper` skill rule and the
hand-maintained (already stale) scoped-AGENTS index; link
architecture docs from Sources of Truth
- .github/AGENTS.md: replace the outdated copied CI command matrix
with a pointer to ci.yml
- crates/AGENTS.md: merge duplicated Testing sections
- ARCHITECTURE.md: resolve the utils->config contradiction (edges are
removed), mark volatile counts as snapshots, fix a bad path
- docs/architecture: add README router; move one-shot plans/trackers
(rebalance-decommission phases, migration-progress ledger, PR
template) to docs/superpowers/plans with archive headers; fix stale
source paths in kept inventories (core/sets.rs, core/pools.rs,
store/mod.rs, startup_* split from #3671)
- docs/operations/tier-ilm-debugging.md: extracted tier debugging
playbook with corrected paths
- scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when
instruction/architecture docs reference nonexistent file paths
- .claude/skills: add tier-debug and arch-checks repo skills;
.gitignore now keeps .claude/skills and docs/operations committable
Verification: ./scripts/check_doc_paths.sh,
./scripts/check_architecture_migration_rules.sh (both pass)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Zhengchao An