Add crates/e2e_test/src/chaos.rs with in-process fault-injection
primitives for a single-node multi-disk RustFS server: take a disk
offline (rename to <dir>.offline), bring it back online, replace a
disk with a fresh empty directory, corrupt an object's erasure shard
(part.* byte flips, xl.meta untouched), and SIGKILL/restart the server
with the same volumes and port.
Add reliability tests on a 4-disk (EC 2+2) topology, verified via
sha256 manifests recorded at write time:
- degraded read/write with one disk offline (incl. multipart object)
- bitrot read-through with two corrupted shards per object
- fresh-disk replacement healed via admin deep heal after a SIGKILL
restart
Also reuse the shared signed_admin_post helper from chaos.rs in the
heal regression suite instead of a duplicated local copy.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(core-storage): fix critical correctness defects from core-storage audit
Fixes verified defects found in a deep audit of the core storage path
(erasure coding, disk persistence, quorum, heal, replication resync):
- ecstore/disk: rewrite live xl.meta atomically (temp+rename) in
delete_versions_internal and write_metadata instead of in-place
truncate, which exposed torn metadata to concurrent readers and
crashes on the DeleteObjects hot path
- ecstore/erasure: allow heal to reconstruct from exactly data_shards
bitrot-verified sources; requiring data_shards+1 made objects
permanently unhealable after losing parity_shards disks
- ecstore/set_disk: direct-memory inline GET applied the erasure
distribution permutation twice (shuffled inputs re-indexed through
distribution), concatenating wrong shards into the response body in
degraded reads; collect from canonical disk-ordered inputs
- ecstore/set_disk: heal now preserves the committed inline layout
instead of recomputing it with a hardcoded unversioned threshold,
which split quorum identity of healed replicas and caused endless
re-heal churn
- ecstore/replication: resync results channel switched from
broadcast(1) to mpsc; a lagged broadcast receiver ended the stats
collector and every subsequent failure went uncounted, letting
failed resyncs be marked completed
- ecstore/replication: ignore an empty persisted resync checkpoint;
resuming with one skipped every object and marked the resync
completed without replicating anything
- ecstore/replication: fix inverted not-found error classification in
replicate_object/replicate_delete logging paths
- ecstore/erasure: guard decode paths against zero block_size or
data_shards from corrupt on-disk metadata (divide-by-zero panic)
- ecstore/disk: os::read_dir no longer consumes the entry limit on
entries it does not return (is_empty_dir misjudgment); create_file
opens with O_TRUNC to avoid stale trailing bytes
- filemeta: treat Some(nil) version id as a null version in
matches_not_strict; disk-loaded headers never store None, so the
mod_time quorum guard for unversioned overwrites never fired and an
interrupted overwrite could displace the committed version in merge
- filemeta: fix msgpack skip lengths for fixext (missed the ext type
byte) and ext16/32 (over-skipped) unknown fields
- filemeta: return FileCorrupt instead of usize underflow when
xl.meta is truncated inside the CRC trailer
- filemeta: surface delete-marker insertion failure in delete_version
instead of reporting success when the data dir is shared
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(replication): drop duplicate cfg(test) etag import from boundary module
The test module already imports content_matches_by_etag locally, so the
top-level cfg(test) import is unused under -D warnings and fails clippy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
fix(ecstore): remove unused cfg(test) import of content_matches_by_etag
The #[cfg(test)] use of content_matches_by_etag at module level is
redundant — the test module already imports it locally at line 369.
This was causing a clippy unused-imports error under -D warnings.
fix(ecstore): purge orphan empty-directory trees on folder delete (#4189)
Empty "phantom" folders — on-disk directory trees with no xl.meta
anywhere — show up in listings as common prefixes but cannot be removed
by any S3 call: DeleteObject on the folder key encodes it to __XLDIR__,
finds no metadata, and returns NotFound, which the API layer masks as a
fake 204 while the tree survives on disk.
Make handle_delete_object attempt a guarded purge when a dir-object key
resolves to NotFound:
- sweep every erasure set of every pool (orphan fragments can sit on
any set, left behind by whichever sets stored the deleted children)
- per set, refuse if ANY disk holds a regular file under the prefix,
so degraded/healable objects are never destroyed
- remove empty directories children-first with non-recursive deletes;
a racing PutObject makes the rmdir fail with DirectoryNotEmpty and
the entry is skipped
This state is unrepresentable on AWS S3 (CommonPrefixes derive only
from real keys; DeleteObject is an idempotent 204), so the purge moves
visible behavior closer to S3. It deliberately goes beyond MinIO, whose
DeleteObject leaves these trees stranded and whose heal only removes
dangling (minority-presence) dirs — a long-standing complaint
(minio/minio#15257, #19516, #18444).
Also move the test-only content_matches_by_etag import in
replication_target_boundary.rs into the cfg(test) module; it failed
cargo clippy -D warnings and blocked the pre-PR gate.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
content_matches_by_etag is only used inside the #[cfg(test)] module, so
the lib-scope import from #4211 fails cargo clippy -D warnings on every
non-test build.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
fix(test): use subset check for capacity dirty scope test to handle global state contamination
The data_movement_put_object_marks_dirty_disks_for_capacity_manager test
asserts exact dirty disk count, but the global dirty scope registry is
process-wide. Other concurrent tests can add entries between the drain
and the assertion, causing flaky failures (8 instead of expected 4).
Use subset assertion to verify the expected disks are present without
being sensitive to entries from other tests.
Remove or consolidate 57 test cases that cannot catch regressions
(literal-constant asserts, construct-then-assert, derived-serde
round-trips, near-duplicate env/getter matrices) in common, config,
iam, madmin, and object-capacity, keeping all wire-format and
error-path guards. Add 13 tests for previously uncovered high-risk
behavior: filemeta version-sort determinism and merge resilience to
garbage headers, zip extraction path-traversal rejection and exact
limit boundaries, JWT tampered-signature rejection, and the bytes
variant of dual-key (rustfs/minio) metadata fallback and precedence.
Test-only change; no production code touched.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(ecstore): remove reachable panics in tiering, replication, and heal paths
- Parse x-amz-expiration leniently in tier PUT responses; any lifecycle
rule on the remote tier bucket returns an RFC1123 date that the previous
ISO8601 unwrap turned into a panic of the ILM transition worker
- Skip invalid user-metadata header values (with a warning) when building
tier and replication PUT headers instead of panicking on non-ASCII input
- Heal: tolerate absent data_dir for delete markers and remote objects
- transition_object: don't unwrap version_id on unversioned buckets when
recording partial writes for offline disks
- Admin server info: use port_or_known_default() so default-port (80/443)
endpoints don't panic is_server_resolvable
- Tier ListObjectsV2 client: decode response body with from_utf8_lossy
- walk_internal: log merge_entry_channels errors instead of dropping them
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(ecstore): fail heal explicitly when data_dir is missing
Address review feedback: unwrap_or_default() silently substituted a nil
UUID when latest metadata lacked data_dir. Delete markers and remote
objects legitimately have no data_dir and skip the data-heal block, but
for a regular object a missing data_dir means corrupt metadata — return
FileCorrupt with a descriptive log instead of building part paths under
a nil UUID directory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Follow-ups to the compatibility harness rework:
- Weekly full sweep now runs as a matrix over both topologies: single
node and the 4-node distributed cluster. Manual dispatch keeps the
test-mode input.
- New mint workflow (weekly + dispatch) runs MinIO Mint against RustFS:
functional suites of real client SDKs and tools (awscli, mc,
aws-sdk-*, minio-*, s3cmd, ...), catching client-specific signing and
streaming edge cases that boto3-only ceph/s3-tests cannot. Report-only
for test failures with a per-suite summary table; fails only when mint
produces no results.
- New scripts/s3-tests/api_coverage.py quantifies API surface coverage
by diffing the s3s S3 trait (at the Cargo.toml pinned revision)
against the methods overridden in impl S3 for FS, distinguishing
NotImplemented defaults from delegating defaults (e.g. post_object).
Current state: 76/101 operations covered (75 overridden, 1 delegated).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Reworks the S3 compatibility test harness for reproducibility and faster
feedback:
- Pin ceph/s3-tests to a fixed commit (S3TESTS_REV, fetch-by-SHA) so the
449-test PR gate is reproducible; previously every run cloned upstream
master, letting test renames or assertion changes break CI silently.
- PR gate (ci.yml s3-implemented-tests): MAXFAIL=0 + XDIST=4 so a single
CI round reports every failure in parallel instead of stopping at the
first one serially.
- Add TEST_SCOPE=all to run.sh to run the entire upstream suite, and
report_compat.py to diff junit results against the classification
lists (regressions, promotion candidates, unclassified tests).
- Rewrite e2e-s3tests.yml: delegate execution to run.sh (single source
of truth; also fixes the broken config generation that left S3_PORT
empty), add a weekly scheduled full sweep that fails only on whitelist
regressions, and fix the multi-node topology to a real distributed
cluster (endpoint-style RUSTFS_VOLUMES) instead of four independent
single-node stores behind a load balancer.
- Docs: rewrite stale .github/s3tests/README.md (marker-era strategy),
update scripts/s3-tests/README.md, fix dead build_testexpr.sh
reference in S3_COMPAT_WORKFLOW.md, drop legacy non_standard_tests.txt.
All 747 classified test names verified present at the pinned revision;
13 upstream tests are currently unclassified and will surface in the
first full-sweep report.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Agent-instruction and architecture docs had drifted from the code:
- CLAUDE.md: slim to commands + pointers; fix wrong claim that
`make pre-commit` is the full pre-PR gate (that is `make pre-pr`);
drop stale pre-#3929 file paths and merged bug narratives
- AGENTS.md: drop dead `rust-refactor-helper` skill rule and the
hand-maintained (already stale) scoped-AGENTS index; link
architecture docs from Sources of Truth
- .github/AGENTS.md: replace the outdated copied CI command matrix
with a pointer to ci.yml
- crates/AGENTS.md: merge duplicated Testing sections
- ARCHITECTURE.md: resolve the utils->config contradiction (edges are
removed), mark volatile counts as snapshots, fix a bad path
- docs/architecture: add README router; move one-shot plans/trackers
(rebalance-decommission phases, migration-progress ledger, PR
template) to docs/superpowers/plans with archive headers; fix stale
source paths in kept inventories (core/sets.rs, core/pools.rs,
store/mod.rs, startup_* split from #3671)
- docs/operations/tier-ilm-debugging.md: extracted tier debugging
playbook with corrected paths
- scripts/check_doc_paths.sh: new guard failing pre-commit/pre-pr when
instruction/architecture docs reference nonexistent file paths
- .claude/skills: add tier-debug and arch-checks repo skills;
.gitignore now keeps .claude/skills and docs/operations committable
Verification: ./scripts/check_doc_paths.sh,
./scripts/check_architecture_migration_rules.sh (both pass)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Speeds up the CI pipeline without changing what is tested:
- Drop the global CARGO_BUILD_JOBS=2 and per-job --jobs 2 flags, which
halved compiler parallelism on the 4-core runners.
- Stop embedding hashFiles(Cargo.lock) in rust-cache shared keys. The
action already fingerprints lockfiles internally; putting the hash in
the key prefix meant any PR that touched Cargo.lock started from a
completely cold cache instead of reusing unchanged dependencies.
- Give the e2e-tests job the full setup action with dependency caching.
Its migration-proof step compiles the e2e_test crate (which pulls in
most of the workspace) and previously did so cold on every run with
no cache, on a 2-core runner.
- Set profile.dev debug = "line-tables-only": keeps usable backtraces
while cutting compile/link time, target-dir size (faster cache
save/restore), and debug-binary artifact upload/download.
- Split fmt + repo-script checks into a compile-free quick-checks job
on ubuntu-latest so contributors get feedback in ~1 minute instead
of after the full test run.
- Collapse the five per-filter migration-proof cargo test reruns into
one filtered nextest invocation; the same tests already run in the
full nextest pass.
- Remove the touch rustfs/build.rs + forced rebuild from the debug
binary jobs (version stamping is irrelevant for e2e binaries) and
the unreachable Windows cross-compile branch in build.yml.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Zhengchao An