gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-28 17:18:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
c3aac2279fcdc1e07972b28a4ff995529dc45b0e
rustfs/docs/testing
T
History
abdullahnah92 c3aac2279f fix(site-replication): keep reverse direction after config broadcast (#5292) 
* fix(site-replication): keep reverse direction after config broadcast

`site-repl-*` rules encode the sender's outbound direction: their
destination ARN names the receiver. `apply_bucket_meta_item` wrote an
incoming rule set verbatim over the receiver's, leaving the receiver with
a rule whose ARN is its own deployment ID. `reconcile_site_replication_bucket_targets`
skips the local peer, so no bucket target can back that ARN and every
object was dropped; the follow-up call reconciled targets only, so
nothing rebuilt the lost reverse rule. Replication went one-directional
after any PutBucketReplication broadcast — the console's Save button,
`mc replicate import`, a metadata import, or `/site-replication/repair`.

Only operator-authored rules now travel between sites; each site owns its
`site-repl-*` rules and rebuilds them from the current peer set.

Four defects kept that invisible or unrecoverable:

- `update_all_targets` discarded target-client build errors silently, and
  `replicate_object` logged the resulting missing-target drop at debug
  while every other failure there logs at error. Both now report.
- `site_replication_rule_complete` never checked that a rule's
  destination named a remote site, so two sites holding identical
  configs — the post-clobber state — passed as in sync.
- `update_service_account` cannot rewrite `parent_user`, and IAM records
  encrypted with a previous root secret decode as "no such account".
  Startup now reconciles the account, reseeding from the secret every
  site-replication bucket target already stores, and refuses the
  delete-then-create sequence when the parent cannot back an account.
  Bucket rules are reconciled too, so an already-broken site heals on
  upgrade.
- A joined site never verified it could reach the initiator, whose
  endpoint is derived from the Host header of the admin request that
  created the topology. The join now probes each peer and reports through
  `initial_sync_error_message`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(site-replication): report unreachable targets and reconcile on a timer

Rule-shape checking cannot see an unreachable peer. A `site-repl-*` rule
can be perfectly formed while the endpoint recorded for its peer is one
this site cannot reach: `update_all_targets` then builds no client and
`replicate_object` drops every object against that ARN, yet the rule set
still reads as correct and the bucket reports in sync.

Each site now reports whether all of its `site-repl-*` rules resolve to a
live target (`SRBucketInfo.replicationTargetsOnline`, read from the
already-resolved client map so the status path stays cheap), and the
status aggregation treats an offline report as a mismatch. The field is
additive and optional: peers that omit it are "unknown", never a fault,
so a mixed-version topology does not flip every bucket to out of sync.

The reconcilers also run on a 10-minute timer instead of at startup only,
so drift is repaired without waiting for a restart. Both are no-ops when
the wiring already matches — the bucket pass compares serialized targets
and the rule set before writing. The tick takes the site-replication
lifecycle lock with a non-blocking try_acquire and skips the round when
an add/remove/endpoint-refresh holds it: those run in phases, and
rebuilding rules between two of them would resurrect what the operation
just tore down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* refactor(site-replication): invert reconcile dependency to satisfy layers

`startup_services.rs` sits in the infra layer and was calling the
reconcilers in `admin::handlers::site_replication`, which is interface —
a reverse dependency that check_layer_dependencies.sh rejects.

Moving the reconcilers down is not viable in this change: they rest on
the site-replication state core (`SiteReplicationState` alone has 107
in-file uses, `load_site_replication_state` 38, the state lock 33), so
relocating it would move ~2000 lines and ~200 call sites through a
bug-fix PR.

Invert the direction instead. A new infra module owns the contract and
the schedule; the admin layer registers its reconciler from
`register_site_replication_route`, which runs while the admin router is
built — `init_startup_http_servers` awaits that before
`init_startup_runtime_services` reconciles, so the hook is always
installed in time. No logic moves and no baseline entry is added: the
dependency genuinely reverses.

The lifecycle guard now wraps both reconcilers in one round rather than
each separately, closing the window between them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(site-replication): harden reconcile per review feedback

Addresses the automated review on #5292.

Security: secret recovery from bucket targets accepted any target carrying
the `site-replicator-0` access key. Bucket targets are writable by anyone
holding `admin:SetBucketTarget`, so such a principal could plant a secret
and have reconciliation recreate the broadly privileged replication
account with it. A target must now name a peer in the persisted state and
point at that peer's recorded endpoint, disagreeing targets abort the
recovery, and only missing/unreadable-account errors may trigger it at
all — a transient store failure no longer rewrites a live account.

Durability: the repair no longer deletes before creating. A readable
account is rebound in place through a new `parent_user` field on
`UpdateServiceAccountOpts`, gated to `site-replicator-0` under
`allow_site_replicator_account` exactly as the account itself is. The
parent also lives in the session-token claims, and
`prepare_service_account_auth` denies the account when the two disagree,
so both move together.

Availability: the reconcile scheduler no longer requires an inline IAM
bootstrap. Deferred IAM recovers in the background with no callback into
the scheduler, which left a recovered node with self-pointing rules until
the next restart. It now starts unconditionally and returns early while
IAM or the object store are unavailable. Its first pass runs inside the
task, so walking every bucket no longer delays startup.

Correctness: an endpoint refresh commits bucket targets and peer state in
separate steps without holding the lifecycle lock, so a tick landing
between them rewrote targets from the stale endpoint; the reconciler now
also skips while any pending marker is set. Rule repair preserves an
operator-authored `role` and clears only sender-owned site-replication
ARNs, matching the merge path.

Hot path: the per-object missing-target message returns to debug. The
condition is reported once per bucket per reconcile pass instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-07-27 22:39:23 +08:00
..
e2e-suite-inventory.md
fix(site-replication): keep reverse direction after config broadcast (#5292)
2026-07-27 22:39:23 +08:00
ecstore-validation-suite-design.md
test(ecstore): add validation suite coverage gates (#4378)
2026-07-07 23:50:12 +08:00
README.md
chore(scripts): add scripts index and archive one-shot scripts (#4822)
2026-07-15 03:43:26 +00:00
security-regressions.md
Merge commit from fork
2026-07-26 07:10:50 +08:00

README.md

RustFS Testing

Owner: backlog#1153 (infra-11). This file is the authoritative home for the test-layer taxonomy, naming conventions, and serial/nextest rules. The event × budget × required-status matrix is owned separately by docs/testing/ci-gates.md (backlog#1149 ci-15); this file links to it rather than duplicating counts, timeouts, or required-check names.

Test taxonomy

RustFS layers its tests from cheap-and-narrow to expensive-and-broad. Higher layers catch what lower layers cannot but cost more wall-clock and setup, so each layer has a clear entry point and a clear "when". Pick the lowest layer that can prove your change; add a higher-layer test only when the behaviour is not observable below it.

Layer What it covers Entry command When it runs
Unit & crate integration Per-crate logic and in-process integration tests, run under nextest cargo nextest run --all --exclude e2e_test (or -p <crate>) Every PR (required)
ecstore black-box Erasure-coded read/write/recovery validation of the ecstore stack scripts/run_ecstore_validation_suite.sh --profile quick Local / release validation (not in CI workflows)
e2e (e2e_test crate) Full server spun up per test, driven over the S3 API cargo nextest run --profile e2e-smoke -p e2e_test PR smoke lane + scheduled full/nightly lanes
s3s-e2e conformance External S3 conformance tool run against a live rustfs server ./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs-e2e-data Per-PR e2e gate (e2e-tests job)
S3 compatibility Third-party suites: ceph/s3-tests (boto3) and MinIO mint (many SDKs) scripts/s3-tests/run.sh (mint: .github/workflows/mint.yml) s3-tests: per-PR gate; mint: scheduled, report-only
Chaos / fault-injection Multi-node, power-loss, and disk-fault harness — (harness planned) Planned — tracked in backlog#1100
Fuzz cargo-fuzz targets over untrusted parsing/validation surfaces ./scripts/fuzz/run.sh (or cd fuzz && cargo +nightly fuzz run <target>) PR smoke + nightly corpus (.github/workflows/fuzz.yml)
Benchmarks Criterion micro/throughput benchmarks cargo bench -p <crate> On-demand / local

The When it runs column is a qualitative pointer only. The authoritative event × timeout × required-status matrix lives in docs/testing/ci-gates.md (ci-15) — do not duplicate its numbers here.

Layer notes:

  • Unit & crate integration — the primary gate. make test wraps cargo nextest run --all --exclude e2e_test; CI runs the same set under the strict ci profile (.config/nextest.toml). Requires cargo-nextest (see Serial execution & nextest profiles).
  • ecstore black-box — scripts/run_ecstore_validation_suite.sh has four profiles (quick / full / destructive / fuzz); quick is the PR-smoke-sized core read/write/recovery pass. It is a local and release-validation tool, not wired into a CI workflow.
  • e2e — each test spawns its own single-node rustfs server on a random port with an isolated temp dir, so the suite is parallel-safe. The e2e-smoke profile is the single PR wiring mechanism; slow/cross-process suites run in scheduled lanes (e2e-repl-nightly, and the reliability group). Full contributor guide: crates/e2e_test/README.md; per-module counts: e2e-suite-inventory.md.
  • s3s-e2e — an external black-box conformance tool installed in CI; locally run it against a freshly built binary with scripts/e2e-run.sh <binary> <data-dir>.
  • S3 compatibility — ceph/s3-tests exercises S3 semantics through boto3 and is gated on a committed allow-list (scripts/s3-tests/implemented_tests.txt); MinIO mint runs many real client SDKs and is report-only until suites pass reliably (see the header of .github/workflows/mint.yml).
  • Chaos / fault-injection — multi-node, power-loss, and disk-fault scenarios are out of scope for this repo's in-tree suites; the harness is tracked in backlog#1100. (Single-node disk-fault e2e tests already live in the e2e crate's e2e-reliability group.)
  • Fuzz — the cargo-fuzz harness is an isolated sub-workspace under fuzz/ (kept out of the root workspace on purpose); see fuzz/README.md for targets and corpus rules.
  • Benchmarks — Criterion benches live under each crate's benches/. They are not a gate; run them locally to compare before/after on a specific crate.

Script inventory: every entry under scripts/ — including the runners above — is indexed in scripts/README.md with its status (ci-gate / dev-tool / archived) and wiring.

Security advisory regression tests

Fixed GHSA advisories map to named, discoverable regression tests. The advisory -> test map lives in docs/testing/security-regressions.md. sec-14 (backlog#1151) formalizes the written admission policy in AGENTS.md.

Naming conventions

Reserved test-name substrings (migration gate)

The migration-critical CI gate selects tests by name substring rather than by module path, so a rename that drops the substring silently thins the gate. These substrings are therefore reserved: keep them in the test name when a test proves migration-critical behaviour.

Substring Guards
data_movement Cross-pool / cross-set object data-movement proofs
rebalance Pool rebalance correctness
decommission Pool decommission correctness
source_cleanup Post-migration source cleanup
delete_marker Delete-marker handling across migration

The gate is enforced by scripts/check_migration_gate_count.sh, which counts the selected tests and fails if the count drops below the committed floor in .config/migration-gate-floor.txt. A deliberate reduction must lower the floor in the same PR, so the change is reviewable in the diff (backlog#1153 infra-12). The substring list above is the same one the gate uses; keep the two in sync when adding a reserved word.

General naming

  • Name a regression test after what it pins: the issue or advisory number (..._regression_test, ..._issue_NNNN_...) or the invariant it protects, so a reviewer can find the guard for a past bug by grepping.
  • e2e lane membership is driven by test-name patterns in .config/nextest.toml (for example the _real_dual_node / _real_single_node markers route replication tests into the nightly lane). Follow the existing marker when adding a test to an established suite; see crates/e2e_test/README.md.
  • Follow the Rust API Guidelines for symbol naming (see AGENTS.md).

Serial execution & nextest profiles

cargo-nextest is the runner. make test requires it and CI installs it; plain cargo test is not a faithful substitute because the two runners execute tests differently.

  • Install: cargo install cargo-nextest --locked, or a prebuilt binary (faster) from https://nexte.st/docs/installation/.
  • Escape hatch: RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test runs the plain cargo test fallback, but the results are not authoritative — serialization semantics differ from CI and [test-groups] do not apply.

Why #[serial] is mostly a no-op under nextest. nextest runs every test in its own process, so serial_test's in-process #[serial] mutex does not serialize tests against each other. The mechanism that actually serializes across nextest's process boundary is a nextest [test-groups] entry with max-threads = 1 (for example ecstore-serial-flaky and e2e-reliability in .config/nextest.toml). Consequences:

  • Do not add #[serial] expecting cross-test isolation under nextest. If two tests genuinely share process/global state or a fixed external resource, serialize them with a [test-groups] entry, or make each test self-isolating (per-test instance context — see backlog#1153 infra-7 / infra-8).
  • A large share of the repo's existing #[serial] markers only affect the cargo test fallback. The serial-debt census and removal plan are tracked in backlog#1153 infra-7.

Profiles (all defined in .config/nextest.toml):

  • default — local runs. Never retries: a red test locally is a real failure to investigate, not noise to retry away.
  • ci — the strict CI gate: global retries = 0 plus a narrowly-scoped quarantine list (retries = 2) for tests with a tracked OPEN flake issue.
  • e2e-smoke — the PR smoke subset of the e2e_test crate (the single wiring mechanism for e2e in PR CI).
  • e2e-repl-nightly — the scheduled slow/cross-process replication lane.

Time control (paused vs real clock)

Time-driven tests should prefer paused time (tokio::time with start_paused and advance) or explicit event synchronization over fixed sleep race windows. The written convention and the docs/testing/time-control.md guide are added by backlog#1153 infra-4.

Coverage

Line coverage is measured weekly, not per-PR, and is non-blocking: it exists for visibility and trend, never as a required check. Per-crate ratchets for the security-critical crates (iam / kms / policy / crypto) build on this baseline later (backlog#1153 infra-6, report-only first).

  • CI: .github/workflows/coverage.yml runs every Sunday and on manual dispatch: cargo llvm-cov nextest --workspace --exclude e2e_test under the ci nextest profile — the same scope and profile as the PR test gate. The per-crate line-coverage table lands in the run's job summary; the lcov + JSON exports are uploaded as a coverage-lcov-<run> artifact kept for 90 days. Scheduled failures open/append the [scheduled-failure] coverage issue via the shared alert action (ci-8).
  • Local: make coverage is the equivalent (slow — instrumented rebuild plus the full suite). It prints the same per-crate table via scripts/coverage_per_crate.py and writes target/llvm-cov/lcov.info and coverage.json.
  • Trend comparison: each run's job summary is the weekly per-crate snapshot — open two runs from the Actions history (workflow "coverage") and compare their tables. For line-level diffs, download the two runs' coverage-lcov-* artifacts and compare the lcov.info files with your lcov tooling of choice.
  • Not measured: doctests (ci.yml runs them uninstrumented; covering them would require a nightly toolchain) and the e2e_test crate (excluded from the unit gate; its lanes are described in the taxonomy above).

Flake policy

A flaky test is one that fails non-deterministically without a corresponding code change. Flakes erode trust in the gate and block tightening required checks, so they are handled on a strict, time-boxed loop.

Retry semantics (source of truth: .config/nextest.toml):

  • The local default profile never retries. A red test on your machine is a real failure to investigate, not noise to paper over.
  • The CI ci profile runs with global retries = 0. A new race must fail on its first occurrence so the first crime scene is never masked.
  • Only tests on the quarantine list get retries = 2, and only under the ci profile. Each quarantine entry MUST link exactly one OPEN issue.
  • JUnit flaky markers are the observable. A quarantined test that passes only after a retry is marked flaky in target/nextest/ci/junit.xml (uploaded as a CI artifact). That marker — not a green check — is how we see a flake is still live.

Lifecycle of a flake:

  1. Discover — a test fails non-deterministically (CI or local), or shows a flaky marker in the JUnit report.
  2. Open an issue within 24h — file/track an issue describing the flake (symptom, suspected cause, affected suite). No silent re-runs.
  3. Quarantine — add the test to the quarantine override block in .config/nextest.toml with a comment linking that OPEN issue. This grants retries = 2 under CI so the flake stops reddening unrelated PRs, while the flaky marker keeps it visible.
  4. Fix or delete within 30 days — make the test robust (then remove the quarantine entry) or delete the test. A quarantine entry may not outlive its fix window; an entry without a live OPEN issue link is a policy violation.

First quarantine members: the two backlog#937 ecstore groups (concurrent_resend_same_part_commits_one_generation and store::bucket::tests::bucket_delete_*).

Reference in New Issue View Git Blame Copy Permalink