2 Commits

Author	SHA1	Message	Date
Zhengchao An	f05a69d51b	test(utils): add rustfs-test-utils crate and shared ECStore bootstrap (#4850) ... * test(utils): add rustfs-test-utils crate, absorb heal/iam ECStore bootstrap backlog#1153 infra-1. The ~50-line "build a real temp-disk ECStore" bootstrap was copy-pasted (and drifting) across the heal and iam integration tests. This adds crates/test-utils (rustfs-test-utils, a dev-dependency-only crate) owning that bootstrap and converts the four copies into thin wrappers: - TestECStoreEnvBuilder: disk_count (default 4), prefix (uuid-suffixed /tmp dir), base_dir (caller-owned dir, e.g. tempfile::TempDir), init_bucket_metadata (default true; the iam bootstrap test opts out to preserve its historical semantics). TestECStoreEnv exposes temp_root/disk_paths/ecstore plus a versioned-bucket helper, and init_tracing() replaces the per-file Once blocks. - All rustfs_ecstore imports stay behind src/ecstore_test_compat.rs, the sanctioned test-compat boundary pattern (mirrors crates/iam/tests/ecstore_test_compat). - heal: heal_integration_test / heal_b5_versioned_regression_test / heal_b920_subquorum_union_test drop their setup_test_env{,_n} copies for heal_env{,_n} wrappers; the tests/storage_api.rs integration surface shrinks to what test bodies still touch. - iam: iam_bootstrap_no_lock_test drops build_local_ecstore; its ecstore_test_compat fixture shrinks to SetupType + update_erasure_type. rg 'async fn setup_test_env' crates/heal crates/iam now returns 0. Scanner's lifecycle tests are deliberately NOT absorbed (gated on ilm-1; 14 of 15 are #[ignore]d today). Net -230 lines. * fix(heal): drop tokio::fs import orphaned by the b920 bootstrap move * fix(heal): drop tokio::fs import orphaned by the b5 bootstrap move	2026-07-15 16:08:30 +08:00
houseme	7001373316	fix(iam): load IAM bootstrap snapshot without namespace locks (#4363) ... * fix(iam): load IAM bootstrap snapshot without namespace locks IAM bootstrap (init_iam_sys -> load_all) read every config object with default ObjectOptions (no_lock=false), so each read acquired a distributed namespace read lock. Lock quorum is counted over cluster nodes and unreachable peers are hard failures, so during a sequential restart the very first read failed with "Quorum not reached: required 2, achieved 0" and IAM could not come up until enough peers' lock RPC surfaces converged - even when the storage read quorum was already satisfiable (rustfs#4304). Extend the startup contract from rustfs#4056 ("startup metadata I/O must not require namespace locks") to the IAM bootstrap path: - Introduce LoadMode {Locked, BootstrapNoLock} and plumb it through the load_all chain (groups, users, policies, mapped policies and their concurrent variants) down to the storage read options. - load_all now performs all reads with no_lock=true; on-line single-object loads via the Store trait keep locked semantics. - Fail-closed behavior is unchanged: any loader error still aborts the whole snapshot load. Safety: config objects are atomic whole-object writes, so a lock-free read only observes an old or a new value; staleness is bounded by the existing periodic IAM reload. Listing (walk) never took namespace locks, and maybe_schedule_lazy_rewrite stays a best-effort background task. Verification: - cargo test -p rustfs-iam --lib (153 passed, incl. new LoadMode tests) - cargo clippy -p rustfs-iam --all-targets - cargo check -p rustfs - make pre-commit Ref: rustfs#4304; tracking rustfs/backlog#884, rustfs/backlog#885 Co-Authored-By: heihutu <heihutu@gmail.com> * test(iam): sequential-restart regression test for lock-free bootstrap Add an integration test reproducing the rustfs#4304 failure mode against a real 4-disk temp-dir ECStore: - Seed IAM group data in single-node mode, then flip the runtime into distributed-erasure mode. new_ns_lock now builds a distributed lock over the set's (empty) lock-client list, so every namespace-locked read fails exactly like a sequential restart with unreachable peers (lock quorum unavailable, storage read quorum healthy). - Assert the locked load_group path fails in that state, while the bulk snapshot load_all (no_lock plumbing from the previous commit) succeeds, and the data survives intact once single-node mode is restored. Reverse-verified: temporarily switching load_all back to the locked mode makes the test fail, so it genuinely guards the contract. Verification: - cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test - cargo test -p rustfs-iam --lib Ref: rustfs#4304; tracking rustfs/backlog#886 Co-Authored-By: heihutu <heihutu@gmail.com> * test(iam): route test ECStore imports through ecstore_test_compat boundary The new integration test imported rustfs_ecstore facade paths directly, tripping three architecture migration rules. Move every ECStore import behind crates/iam/tests/ecstore_test_compat/mod.rs (the sanctioned test-compat pattern), and register that module as a reviewed test-only global-facade boundary in check_architecture_migration_rules.sh: the sequential-restart regression test needs api::global::update_erasure_type to flip into distributed-erasure mode for lock-quorum fault injection. Verification: - ./scripts/check_architecture_migration_rules.sh - cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test - make pre-commit Co-Authored-By: heihutu <heihutu@gmail.com> * feat(server): expose readiness blocking reason + rolling-restart runbook Operators hitting the rustfs#4304 sequential cold start could not tell from the outside why a node stayed unavailable. Three additions: - The readiness gate's 503 now names the blocking dependency in both the body ("Service not ready: waiting for storage_quorum") and a new x-rustfs-readiness-pending header (storage_quorum | iam | startup_finalization), derived from the current startup stage. /health/ready already returned details + degradedReasons; this covers the plain S3 requests that hit the gate. - IAM bootstrap retry logs now carry an actionable `hint` field that classifies the failure (storage read quorum vs lock quorum vs uninitialized metadata) instead of only echoing the storage error. - New docs/operations/rolling-restart.md runbook: correct rolling restart procedure, sequential cold-start expectations (degraded -> auto-recovery), readiness signal reference, and RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS guidance. Verification: - cargo test -p rustfs --lib -- hint_tests service_not_ready readiness_pending - make pre-commit Ref: rustfs#4304; tracking rustfs/backlog#887 Co-Authored-By: heihutu <heihutu@gmail.com> * upgrade deps version and improve import * feat(iam): notification-path cache refreshes read without namespace locks (#4368) P3 step 1 of rustfs/backlog#884 (scoped down from full MinIO readConfig alignment after review): cross-node notification handlers (group/policy/policy-mapping/user) refresh the local IAM cache with single-object reads that previously took distributed namespace read locks. These refreshes are asynchronous, best-effort, and already stale-tolerant (the periodic reload converges them), so a node-counted lock quorum failure or lock RPC hiccup on a peer must not fail them — the same rationale as the lock-free bootstrap load_all (rustfs#4304). - Store trait: add load_user_no_lock / load_group_no_lock / load_policy_doc_no_lock / load_mapped_policy_no_lock with defaults forwarding to the locked variants, so existing implementations and test mocks keep their behavior. - ObjectStore overrides them via the existing LoadMode::BootstrapNoLock plumbing. Deletions triggered by the handlers keep locked writes. - manager.rs: the four *_notification_handler paths (8 call sites) switch to the lock-free variants. - Integration test: while the lock quorum is unavailable (DistErasure with empty lockers), load_group_no_lock must succeed exactly where the locked load_group fails. Request-path loads (check_key, verify_temp_user_persistence) and admin write-then-reload paths intentionally stay locked: load_user_identity embeds expiry deletions, so those need the side-effect extraction tracked in rustfs/backlog#884 before going lock-free. Verification: - cargo test -p rustfs-iam --lib (156 passed) - cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test - make pre-commit Ref: rustfs/backlog#884, rustfs#4304 Co-authored-by: heihutu <heihutu@gmail.com> * fix(server): drop unused iam_bootstrap_failure_hint import in tests The hint tests live in their own hint_tests module with a local import; the stale re-import in mod tests failed clippy's -D warnings on the Test and Lint CI variants. Verification: - cargo clippy -p rustfs --all-targets Co-Authored-By: heihutu <heihutu@gmail.com> * fix --------- Co-authored-by: heihutu <heihutu@gmail.com>	2026-07-07 20:07:09 +08:00