* fix(site-replication): keep reverse direction after config broadcast
`site-repl-*` rules encode the sender's outbound direction: their
destination ARN names the receiver. `apply_bucket_meta_item` wrote an
incoming rule set verbatim over the receiver's, leaving the receiver with
a rule whose ARN is its own deployment ID. `reconcile_site_replication_bucket_targets`
skips the local peer, so no bucket target can back that ARN and every
object was dropped; the follow-up call reconciled targets only, so
nothing rebuilt the lost reverse rule. Replication went one-directional
after any PutBucketReplication broadcast — the console's Save button,
`mc replicate import`, a metadata import, or `/site-replication/repair`.
Only operator-authored rules now travel between sites; each site owns its
`site-repl-*` rules and rebuilds them from the current peer set.
Four defects kept that invisible or unrecoverable:
- `update_all_targets` discarded target-client build errors silently, and
`replicate_object` logged the resulting missing-target drop at debug
while every other failure there logs at error. Both now report.
- `site_replication_rule_complete` never checked that a rule's
destination named a remote site, so two sites holding identical
configs — the post-clobber state — passed as in sync.
- `update_service_account` cannot rewrite `parent_user`, and IAM records
encrypted with a previous root secret decode as "no such account".
Startup now reconciles the account, reseeding from the secret every
site-replication bucket target already stores, and refuses the
delete-then-create sequence when the parent cannot back an account.
Bucket rules are reconciled too, so an already-broken site heals on
upgrade.
- A joined site never verified it could reach the initiator, whose
endpoint is derived from the Host header of the admin request that
created the topology. The join now probes each peer and reports through
`initial_sync_error_message`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): report unreachable targets and reconcile on a timer
Rule-shape checking cannot see an unreachable peer. A `site-repl-*` rule
can be perfectly formed while the endpoint recorded for its peer is one
this site cannot reach: `update_all_targets` then builds no client and
`replicate_object` drops every object against that ARN, yet the rule set
still reads as correct and the bucket reports in sync.
Each site now reports whether all of its `site-repl-*` rules resolve to a
live target (`SRBucketInfo.replicationTargetsOnline`, read from the
already-resolved client map so the status path stays cheap), and the
status aggregation treats an offline report as a mismatch. The field is
additive and optional: peers that omit it are "unknown", never a fault,
so a mixed-version topology does not flip every bucket to out of sync.
The reconcilers also run on a 10-minute timer instead of at startup only,
so drift is repaired without waiting for a restart. Both are no-ops when
the wiring already matches — the bucket pass compares serialized targets
and the rule set before writing. The tick takes the site-replication
lifecycle lock with a non-blocking try_acquire and skips the round when
an add/remove/endpoint-refresh holds it: those run in phases, and
rebuilding rules between two of them would resurrect what the operation
just tore down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor(site-replication): invert reconcile dependency to satisfy layers
`startup_services.rs` sits in the infra layer and was calling the
reconcilers in `admin::handlers::site_replication`, which is interface —
a reverse dependency that check_layer_dependencies.sh rejects.
Moving the reconcilers down is not viable in this change: they rest on
the site-replication state core (`SiteReplicationState` alone has 107
in-file uses, `load_site_replication_state` 38, the state lock 33), so
relocating it would move ~2000 lines and ~200 call sites through a
bug-fix PR.
Invert the direction instead. A new infra module owns the contract and
the schedule; the admin layer registers its reconciler from
`register_site_replication_route`, which runs while the admin router is
built — `init_startup_http_servers` awaits that before
`init_startup_runtime_services` reconciles, so the hook is always
installed in time. No logic moves and no baseline entry is added: the
dependency genuinely reverses.
The lifecycle guard now wraps both reconcilers in one round rather than
each separately, closing the window between them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix(site-replication): harden reconcile per review feedback
Addresses the automated review on #5292.
Security: secret recovery from bucket targets accepted any target carrying
the `site-replicator-0` access key. Bucket targets are writable by anyone
holding `admin:SetBucketTarget`, so such a principal could plant a secret
and have reconciliation recreate the broadly privileged replication
account with it. A target must now name a peer in the persisted state and
point at that peer's recorded endpoint, disagreeing targets abort the
recovery, and only missing/unreadable-account errors may trigger it at
all — a transient store failure no longer rewrites a live account.
Durability: the repair no longer deletes before creating. A readable
account is rebound in place through a new `parent_user` field on
`UpdateServiceAccountOpts`, gated to `site-replicator-0` under
`allow_site_replicator_account` exactly as the account itself is. The
parent also lives in the session-token claims, and
`prepare_service_account_auth` denies the account when the two disagree,
so both move together.
Availability: the reconcile scheduler no longer requires an inline IAM
bootstrap. Deferred IAM recovers in the background with no callback into
the scheduler, which left a recovered node with self-pointing rules until
the next restart. It now starts unconditionally and returns early while
IAM or the object store are unavailable. Its first pass runs inside the
task, so walking every bucket no longer delays startup.
Correctness: an endpoint refresh commits bucket targets and peer state in
separate steps without holding the lifecycle lock, so a tick landing
between them rewrote targets from the stale endpoint; the reconciler now
also skips while any pending marker is set. Rule repair preserves an
operator-authored `role` and clears only sender-owned site-replication
ARNs, matching the merge path.
Hot path: the per-object missing-target message returns to debug. The
condition is reported once per bucket per reconcile pass instead.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
* fix(sse): surface unconfigured managed SSE as 400, fix POST SSE-S3 e2e
Managed SSE (SSE-S3 / bucket-default) on a server without KMS and without RUSTFS_SSE_S3_MASTER_KEY fails closed by design since #3564, but surfaced as 500 InternalError. Map the misconfiguration to InvalidRequest (400) and seed the local SSE master key in the anonymous POST-object SSE e2e tests; align the missing-from-policy test with the MinIO-compatible SSE field exemption (s3s#608). Re-admit the tests to the e2e-full profile (rustfs#4844).
* fix(auth): POST-object lock fields and signed metadata=true listing 403s
Two authorization surfaces returned 403 for allowed requests (rustfs#4845): the PutObject access hook demanded PutObjectRetention/PutObjectLegalHold IAM actions for POST-object form uploads whose lock fields are governed by the validated POST policy, and the metadata=true listing route never resolved SigV4-verified credentials into ReqInfo so signed requests were evaluated as anonymous. Skip the PUT-header lock actions for POST form uploads and resolve credentials in the metadata route; re-admit the four e2e tests to the e2e-full profile.
fix(sse): surface unconfigured managed SSE as 400, fix POST SSE-S3 e2e
Managed SSE (SSE-S3 / bucket-default) on a server without KMS and without RUSTFS_SSE_S3_MASTER_KEY fails closed by design since #3564, but surfaced as 500 InternalError. Map the misconfiguration to InvalidRequest (400) and seed the local SSE master key in the anonymous POST-object SSE e2e tests; align the missing-from-policy test with the MinIO-compatible SSE field exemption (s3s#608). Re-admit the tests to the e2e-full profile (rustfs#4844).
An extracted entry whose tar header mtime is 0 was stored with
mod_time = UNIX_EPOCH, which xl.meta encodes as 0 nanos (= no
mod_time). On read-back the version failed valid() and parsing fell
into the legacy rmp_serde fallback, so every read of the object
returned 500 (invalid type: integer 0, expected an OffsetDateTime).
Treat mtime 0 as unset (tar convention) and fall back to the upload
time. Also fix two test-side issues uncovered behind the 500: the pax
fixture must use a ustar header for its XHeader entry, and the SSE-S3
extract tests must provision RUSTFS_SSE_S3_MASTER_KEY. Re-admit the 19
quarantined tests to the e2e-full merge gate.
Fixes#4842
test(e2e): add drivesPerNode and 2-pool cluster harness topology (backlog #1325)
Extend the e2e cluster harness so tests can declare per-node multiple drives (drivesPerNode) and a two-pool topology, alongside a smoke suite that boots such a cluster and round-trips a PUT/GET.
A new `ClusterTopology` describes node_count, drives_per_node, and pool membership, and `RustFSTestClusterEnvironment::with_topology` builds the matching on-disk drive directories and `RUSTFS_VOLUMES` string. `new(node_count)` now delegates to a single-pool single-drive topology, so existing single-drive cluster tests are byte-for-byte unchanged. `ClusterNode` gains `data_dirs` (all drives, `data_dirs[0] == data_dir`) and `pool_idx`; multi-drive/multi-pool clusters automatically add `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true` because their drives share the same temp filesystem.
The `RUSTFS_VOLUMES` assembly matches the two forms the server parser accepts on a single localhost machine (verified against ecstore `DisksLayout::from_volumes`): a single pool is the explicit enumeration of every `(node, drive)` endpoint (no ellipses, one legacy DistErasure pool), while multiple pools each contribute one ellipses argument `http://<addr><node-base>/drive{0...N-1}`. A pool argument is a single URL template, so it cannot enumerate multiple distinct-port hosts; a pool striped across several localhost nodes would need a host ellipses that forces a shared on-disk path and collides physically. The topology validator therefore requires every pool in a multi-pool layout to own exactly one node and requires drives_per_node >= 2 (the parser rejects a single-drive `drive{0...0}` ellipses pool). Genuine multi-node pools need real multi-host infrastructure and are deferred to the nightly cluster lane (backlog #1313/#1314).
Unit tests assert the volumes-string layout for single-pool single-drive (backward compatible), single-pool multi-drive (8 explicit endpoints), and two-pool (one ellipses arg per pool), plus the validator rejections. The smoke suite `cluster_multidrive_pool_test` boots a 4-node x 2-drive single pool and a 2-node/2-pool cluster, asserts the volumes layout, and round-trips a PUT/GET using the harness readiness handshake (no fixed sleeps). It joins the six existing RustFSTestClusterEnvironment suites excluded from the merge-gate `e2e-full` profile so it runs in the nightly 4-node lane.
Network fault injection (toxiproxy / socket proxy) and 5GiB large-object budgets remain out of scope for this block.
First systematic HTTP e2e batch for the admin management surface
(backlog#1154 peri-2): full user -> canned-policy -> attach ->
service-account lifecycle with data-plane effect assertions (the
credential really gains and loses S3 access), plus a non-admin 403
probe per management endpoint targeting a different principal so
deny_only self-access semantics do not mask the gate. Wired into the
e2e-smoke nextest profile (ci-4 mechanism).
Pin the console listener's wire contract with a real binary on a random
port (backlog#1154 peri-4): unauthenticated version/license endpoints
answer complete JSON without credential material, the SPA prefix always
dispatches to the console static handler (never the S3 API), the admin
API and S3 root on the console listener stay 403 for anonymous callers,
and a console-disabled listener serves no console surface. Wired into
the e2e-smoke nextest profile (ci-4 mechanism).
Prove the swap-certificates-without-restart promise over real TLS
handshakes (backlog#1154 peri-5): new connections pick up the rotated
certificate within the reload interval, a session opened under the old
certificate survives the swap, and garbage material is fail-safe (the
old certificate keeps serving, the failure leaves a tls_reload_failed
log event, the process stays up). Wired into the e2e-smoke nextest
profile (ci-4 mechanism).
* ci: add e2e-full merge-gate job (single-node full e2e via nextest)
Adds the [profile.e2e-full] nextest profile and a matching e2e-full job in
ci.yml (push main + merge_group + workflow_dispatch) that runs the
never-automated user-visible e2e suites (KMS, object_lock, multipart_auth,
quota, checksum, encryption, security-boundary, ...) the fast PR e2e-smoke
subset skips.
The filter is the whole e2e_test crate minus the sets other lanes own:
protocols:: (ci-6/ci-7), the 6 RustFSTestClusterEnvironment cluster suites
(ci-7 nightly), replication_extension_test (repl-1's smoke + repl-nightly
lanes), and #[ignore]d tests (ci-13). The 4-disk reliability tests are
serialized, mirroring the ci profile. Reuses the downloaded debug binary and
uploads junit. backlog#1149 ci-5.
* ci: exclude characterized known-failures from e2e-full with tracked issues
First-ever automated run of the full suite (dispatch 29381309848: 341 ran /
32 failed / 460s) surfaced five real product-bug families, each now tracked:
rustfs#4842 (extract 500s, mtime=0 OffsetDateTime), rustfs#4843 (path-limit
vs ignore-errors), rustfs#4844 (anonymous POST SSE-S3 500), rustfs#4845
(object-lock POST + list metadata=true 403), rustfs#4846 (lock quorum
misclassified as timeout under load). Deterministic failures cannot be
retried away, so each family is excluded with its OPEN issue cited and the
fixing PR contract to delete the exclusion — passing negative-path siblings
stay in as regression guards.
The header-SigV4 (sec-1), presigned-URL (sec-2), and admin-gate (sec-4) negative
auth-rejection e2e suites merged earlier but only presigned_negative was actually
selected by any CI profile; negative_sigv4_test and admin_auth_test compiled and
sat unrun. Add both to the e2e-smoke default-filter so all three attacker-facing
S3 auth-rejection suites execute on every PR. They already meet the smoke
admission criteria (RustFSTestEnvironment, random ports, parallel-safe, no
#[ignore], no feature gates), so this is a pure filterset change — the single
e2e-in-CI wiring mechanism (backlog#1149 ci-4), not a new job.
Because the filter selects by module name, a rename or deletion could silently
drop a suite out of the security gate with no CI signal. Add
scripts/check_security_smoke_count.sh (infra-12 count-floor mechanism): it lists
what the e2e-smoke profile selects and fails if the count of security
auth-rejection tests drops below the committed floor in
.config/security-smoke-floor.txt (16). Invoked from the e2e-tests job, before the
smoke run, so the nextest list compiles the binaries the run reuses.
The GHSA-3p3x FTPS/WebDAV constant-time e2e (protocols::test_protocol_core_suite)
stays out by topology: it binds fixed ports, needs the ftps,webdav features, and
is #[serial], so it cannot join the random-port default-feature smoke profile as
a filterset change (global ruling G5). Its GHSA-r5qv sibling is a unit test that
already runs in the default CI pass. docs/testing/security-regressions.md now
carries the full CI-execution map and flags the GHSA-3p3x e2e CI-lane gap as a
ci-domain follow-up.
Refs: backlog#1151 (sec-5)
Convert the two reliant lifecycle expiry tests from #[ignore] + a hardcoded
localhost:9000 server to self-managed RustFSTestEnvironment tests (random port,
isolated temp dir) and wire them into the PR e2e-smoke subset.
Time control is chosen per test and documented in-module:
- test_lifecycle_expiry_backdated_mtime: mod_time back-dating via the internal
source-replication backdoor (new put_object_with_backdated_mtime helper) so a
Days=1 rule is already due, with RUSTFS_ILM_PROCESS_TIME=1 shrinking the
rounding boundary. Asserts the backdated matching-prefix object expires and a
non-matching-prefix object with the same backdating survives (isolates the
prefix filter). Proves the backdoor does not trip the replication gate.
- test_lifecycle_versioned_current_version_expiry_creates_delete_marker:
RUSTFS_ILM_DEBUG_DAY_SECS=2 (ilm-5) accelerates the day length; asserts a
latest delete marker is created, the original data version is retained and
still readable by version id.
- test_lifecycle_zero_day_expiry: Days=0 immediate expiry; matching object
deleted, non-matching survives.
All three use RUSTFS_SCANNER_CYCLE=1 so the scanner runs every second; tests poll
for the terminal state instead of sleeping fixed wall-clock. Ignore count for
reliant/lifecycle.rs goes 2 -> 0.
CI wiring: added a distinct 'test(/^reliant::lifecycle::/)' clause to the
existing profile.e2e-smoke default-filter in .config/nextest.toml (the single
sanctioned e2e wiring mechanism per crates/e2e_test/README.md; no new e2e job).
Refs rustfs/backlog#1148 (ilm-3), rustfs/backlog#1155
test(e2e): negative presigned-URL SigV4 suite (backlog#1151 sec-2)
Add crates/e2e_test/src/presigned_negative_test.rs, the query-string
SigV4 sibling of the header-SigV4 suite (sec-1, #4708). Presigned URLs
were only exercised on the happy path, so nothing pinned our end-to-end
enforcement of expiry or query-signature verification.
Against a live single-node RustFSTestEnvironment (random port, isolated
temp dir), the suite asserts HTTP status + S3 error <Code> for:
- expired presigned GET -> 403 AccessDenied (Request has expired)
- tampered X-Amz-Signature -> 403 SignatureDoesNotMatch
- wrong secret key -> 403 SignatureDoesNotMatch
- tampered target key (swapped after signing) -> 403 SignatureDoesNotMatch
- tampered presigned PUT -> 403 SignatureDoesNotMatch + object not stored
- positive controls: valid presigned GET (body match) and PUT (HEAD verify)
Expiry is controlled without real waiting via the AWS SDK presigner's
start_time (sign as of one hour ago with a 60s window). s3s checks
expiry before the signature, so the expired case surfaces AccessDenied.
Wired into the e2e-smoke nextest profile (fast, single-node, no external
deps) so it runs on every PR per the crates/e2e_test/README.md admission
criteria.
Refs rustfs/backlog#1151 (sec-2), rustfs/backlog#1155.
* fix(replication): allow loopback replication targets under an explicit test opt-in
Commit 5c7c757a3 (#4712) activated the previously-dormant replication e2e
suite (they had never run anywhere). All 9 fast tests then failed on main
because the SSRF egress guard rejects the 127.0.0.1 targets the e2e harness
configures: `target endpoint is not allowed: outbound URL host '127.0.0.1'
is not allowed: loopback address`. The whole harness runs on loopback, so
every replication test hit this before reaching its actual assertion.
Loopback is a genuine SSRF vector and must stay rejected in production, so
this does not relax the guard. Instead `validate_replication_target_endpoint`
gains an off-by-default opt-in (`RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET`)
that re-enables loopback targets (127.0.0.1 / ::1 / localhost) for single-host
multi-instance dev and the e2e harness. Private addresses stay unconditionally
allowed as before; the opt-in does not widen into link-local or the cloud
metadata endpoint. The e2e harness sets the env for every server it spawns
(single-node and cluster paths), overridable via extra_env.
Verified end-to-end: all 9 previously-failing replication_extension_test
smoke tests pass against a locally built binary. New unit tests in
bucket_target_sys pin the matrix — public/private always allowed, loopback
gated on the opt-in in both IP and hostname forms, and metadata/link-local
still rejected even with the opt-in on.
Refs: backlog#1147
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(replication): rename optin -> opt_in to satisfy typos check
Pure rename of three unit-test function names; no behaviour change.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
ci: pull replication tests out of e2e-smoke — loopback targets fail every PR
repl-1 (#4712) added 20 bucket-replication admin-path tests to the
e2e-smoke PR lane. They set a remote replication target at a loopback
endpoint (127.0.0.1, a second local server), which RustFS's target SSRF
guard rejects by default ('outbound URL host 127.0.0.1 is not allowed:
loopback address'). repl-1's local verification was inconclusive under
machine load, so this shipped broken and failed End-to-End Tests on
every PR based on post-repl-1 main (#4707 etc).
Move all replication e2e tests to the e2e-repl-nightly lane (now the full
replication_extension_test set, no allowlist split) to un-block PRs. The
nightly job needs loopback-target-allow server config for these to pass;
tracked as a repl-1 follow-up (backlog#1147). e2e-smoke returns to the
17 functional modules from ci-4 (63 tests).
Adds an S3-API-level e2e regression net proving that a large-object GET on a
degraded EC set never returns a silently truncated body under a full
Content-Length -- the historical "unexpected EOF" bug fixed on main by
rustfs#4594 (short-body GetObject stream -> UnexpectedEof), rustfs#4560
(in-place per-part legacy degradation for the lazy multipart reader), and
rustfs#4585 (DARE package-boundary truncation detection). Those fixes each
ship a *unit* regression; this covers the layer they do not -- the full HTTP
GET path streaming a real body reconstructed from real on-disk EC shards.
New file crates/e2e_test/src/degraded_read_eof_regression_test.rs, single-node
4-disk (EC 2+2) DiskFaultHarness, three scenarios:
(a) one disk offline: a 6 MiB single object (>=2 EC stripes) and a 3x5 MiB
multipart object GET back byte-identical with the correct Content-Length.
(b) mid-stream bitrot within quorum: 2 of 4 shards corrupted mid-file on a
large multipart object still reconstructs the full, hash-matching body.
(c) beyond read quorum (the heart of the net): 3 of 4 shards corrupted
mid-file -- the read MUST fail cleanly (non-2xx or a mid-stream body
error), NEVER close with a truncated body under the full Content-Length.
The shared get_checked() helper panics on the forbidden outcome (a clean 2xx
whose collected body is shorter than the advertised Content-Length), so the
truncation bug can never be silently tolerated.
CI placement: these spawn a 4-disk server per test and are resource-heavy, so
they stay OUT of the fast PR e2e-smoke filter. A new e2e-reliability nextest
test-group (max-threads=1) serializes them (and the existing
reliability_disk_fault_test) across nextest's process boundary; ci-7's nightly
full-e2e run picks them up. Visible via `cargo nextest list -p e2e_test`.
Refs rustfs/backlog#1150 (dist-13), rustfs/backlog#1155, rustfs#4594,
rustfs#4560, rustfs#4585, rustfs#2955.
Activate the 36 dormant replication e2e tests in
crates/e2e_test/src/replication_extension_test.rs (zero ran anywhere before).
Split via the ci-4 nextest profile mechanism, no hand-rolled cargo-test lane:
- PR smoke (profile.e2e-smoke, existing e2e-tests job): the 20 fast
bucket-replication tests (target-registration / replication-check / list /
remove / delete admin paths) that validate config synchronously and never
wait for async convergence. Each spawns its own single-node rustfs server(s)
on random ports with isolated temp dirs, so parallel-safe by construction
(serial_test's #[serial] is a no-op under nextest's process-per-test model;
no test-group needed).
- Nightly (profile.e2e-repl-nightly + .github/workflows/e2e-replication-nightly.yml):
the remaining 16 = 6 slow data-plane tests + 9 _real_dual_node + 1
_real_single_node. Defined as 'replication module MINUS the PR allowlist' so
new replication tests default to nightly and are never silently unrun.
The nightly workflow builds the binary once, installs awscurl so the STS
dual-node test runs (skips gracefully with a visible log line otherwise), and
routes scheduled failures through .github/actions/schedule-failure-issue
(ci-8). Explicit division of labor with ci-5 e2e-full: these run only here.
Counts (cargo nextest list): e2e-smoke 83 (63 + 20), e2e-repl-nightly 16.
Docs updated: e2e-suite-inventory.md, e2e_test/README.md.
Refs backlog#1147 repl-1, backlog#1155.
ci: quarantine walk_dir stall-budget flake under the ci profile (rustfs#4690)
First application of the flake policy from docs/testing/README.md: the
test failed on a zero-Rust-diff PR (#4674, run 29099382470) — timing
windows in the stall-budget accounting stretch under CI load. retries=2
under profile.ci only; local default profile still never retries.
Tracked by rustfs#4690 (30-day fix-or-delete deadline).
ci: run e2e smoke subset via nextest e2e-smoke profile (backlog#1149 ci-4)
The e2e-tests job previously ran a single test (delete_marker_migration
_semantics) out of ~400 in the e2e_test crate. Define a nextest
profile.e2e-smoke whose default-filter selects 17 fast single-node
dependency-free modules (63 tests) and run it in the job, reusing the
downloaded debug binary. The profile is the single wiring mechanism for
e2e tests in CI; admission criteria live in crates/e2e_test/README.md,
and docs/testing/e2e-suite-inventory.md records authoritative per-module
counts from cargo nextest list.
test(ci): add strict nextest ci profile with quarantine + flake policy
Formalize the existing ecstore-serial-flaky mechanism into a strict CI gate
(ci-10, absorbs infra-15; backlog#1149).
- .config/nextest.toml: add [profile.ci] with global retries=0 (never mask a
new race's first occurrence), fail-fast=false, and JUnit output at
target/nextest/ci/junit.xml. Add a quarantine section where flaky tests get
retries=2 under the ci profile only; each entry links one OPEN issue. First
members are the two backlog#937 ecstore groups
(concurrent_resend_same_part_commits_one_generation and
store::bucket::tests::bucket_delete_*), which keep their existing
ecstore-serial-flaky test-group serialization. Local default profile still
never retries.
- .github/workflows/ci.yml: run the main test step with --profile ci and upload
the JUnit report (if: always(), 3-day retention, run-number in name). The
migration-proof step stays on the default profile to avoid clobbering the ci
JUnit artifact (its tests are not quarantined).
- docs/testing/README.md: new skeleton (owned by backlog#1153 infra-11) holding
the flake policy: discover -> open issue within 24h -> quarantine with issue
link -> fix or delete within 30 days. AGENTS.md points to it.
Refs: rustfs/backlog#1149, rustfs/backlog#937, rustfs/backlog#1155
Two ecstore test groups pass in isolation but flake under the full parallel
nextest suite, producing CI noise on every in-flight PR (backlog #937):
- store::bucket::tests::bucket_delete_* race make_bucket into
InsufficientWriteQuorum because they share global disk-registry/lock-client
state with other concurrently running tests.
- bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
exceeds its already-maxed 60s lock-acquire deadline only when the suite
saturates disk I/O.
serial_test's #[serial] does not serialize these across nextest's per-test
process boundary. Add a nextest test-group (max-threads = 1) that matches both
groups so they run serialized, removing the load-induced flake. No test or
production code changes; long-term global-singleton isolation is tracked
separately.
Co-authored-by: heihutu <heihutu@gmail.com>
houseme