gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-31 01:09:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(iam): load IAM bootstrap snapshot without namespace locks (#4363)

Browse Source 
* fix(iam): load IAM bootstrap snapshot without namespace locks

IAM bootstrap (init_iam_sys -> load_all) read every config object with
default ObjectOptions (no_lock=false), so each read acquired a
distributed namespace read lock. Lock quorum is counted over cluster
nodes and unreachable peers are hard failures, so during a sequential
restart the very first read failed with
"Quorum not reached: required 2, achieved 0" and IAM could not come up
until enough peers' lock RPC surfaces converged - even when the storage
read quorum was already satisfiable (rustfs#4304).

Extend the startup contract from rustfs#4056 ("startup metadata I/O must
not require namespace locks") to the IAM bootstrap path:

- Introduce LoadMode {Locked, BootstrapNoLock} and plumb it through the
  load_all chain (groups, users, policies, mapped policies and their
  concurrent variants) down to the storage read options.
- load_all now performs all reads with no_lock=true; on-line
  single-object loads via the Store trait keep locked semantics.
- Fail-closed behavior is unchanged: any loader error still aborts the
  whole snapshot load.

Safety: config objects are atomic whole-object writes, so a lock-free
read only observes an old or a new value; staleness is bounded by the
existing periodic IAM reload. Listing (walk) never took namespace locks,
and maybe_schedule_lazy_rewrite stays a best-effort background task.

Verification:
- cargo test -p rustfs-iam --lib (153 passed, incl. new LoadMode tests)
- cargo clippy -p rustfs-iam --all-targets
- cargo check -p rustfs
- make pre-commit

Ref: rustfs#4304; tracking rustfs/backlog#884, rustfs/backlog#885

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(iam): sequential-restart regression test for lock-free bootstrap

Add an integration test reproducing the rustfs#4304 failure mode against
a real 4-disk temp-dir ECStore:

- Seed IAM group data in single-node mode, then flip the runtime into
  distributed-erasure mode. new_ns_lock now builds a distributed lock
  over the set's (empty) lock-client list, so every namespace-locked
  read fails exactly like a sequential restart with unreachable peers
  (lock quorum unavailable, storage read quorum healthy).
- Assert the locked load_group path fails in that state, while the
  bulk snapshot load_all (no_lock plumbing from the previous commit)
  succeeds, and the data survives intact once single-node mode is
  restored.

Reverse-verified: temporarily switching load_all back to the locked
mode makes the test fail, so it genuinely guards the contract.

Verification:
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- cargo test -p rustfs-iam --lib

Ref: rustfs#4304; tracking rustfs/backlog#886

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(iam): route test ECStore imports through ecstore_test_compat boundary

The new integration test imported rustfs_ecstore facade paths directly,
tripping three architecture migration rules. Move every ECStore import
behind crates/iam/tests/ecstore_test_compat/mod.rs (the sanctioned
test-compat pattern), and register that module as a reviewed test-only
global-facade boundary in check_architecture_migration_rules.sh: the
sequential-restart regression test needs api::global::update_erasure_type
to flip into distributed-erasure mode for lock-quorum fault injection.

Verification:
- ./scripts/check_architecture_migration_rules.sh
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- make pre-commit

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(server): expose readiness blocking reason + rolling-restart runbook

Operators hitting the rustfs#4304 sequential cold start could not tell
from the outside why a node stayed unavailable. Three additions:

- The readiness gate's 503 now names the blocking dependency in both the
  body ("Service not ready: waiting for storage_quorum") and a new
  x-rustfs-readiness-pending header (storage_quorum | iam |
  startup_finalization), derived from the current startup stage.
  /health/ready already returned details + degradedReasons; this covers
  the plain S3 requests that hit the gate.
- IAM bootstrap retry logs now carry an actionable `hint` field that
  classifies the failure (storage read quorum vs lock quorum vs
  uninitialized metadata) instead of only echoing the storage error.
- New docs/operations/rolling-restart.md runbook: correct rolling
  restart procedure, sequential cold-start expectations (degraded ->
  auto-recovery), readiness signal reference, and
  RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS guidance.

Verification:
- cargo test -p rustfs --lib -- hint_tests service_not_ready readiness_pending
- make pre-commit

Ref: rustfs#4304; tracking rustfs/backlog#887

Co-Authored-By: heihutu <heihutu@gmail.com>

* upgrade deps version and improve import

* feat(iam): notification-path cache refreshes read without namespace locks (#4368)

P3 step 1 of rustfs/backlog#884 (scoped down from full MinIO readConfig
alignment after review): cross-node notification handlers
(group/policy/policy-mapping/user) refresh the local IAM cache with
single-object reads that previously took distributed namespace read
locks. These refreshes are asynchronous, best-effort, and already
stale-tolerant (the periodic reload converges them), so a node-counted
lock quorum failure or lock RPC hiccup on a peer must not fail them —
the same rationale as the lock-free bootstrap load_all (rustfs#4304).

- Store trait: add load_user_no_lock / load_group_no_lock /
  load_policy_doc_no_lock / load_mapped_policy_no_lock with defaults
  forwarding to the locked variants, so existing implementations and
  test mocks keep their behavior.
- ObjectStore overrides them via the existing LoadMode::BootstrapNoLock
  plumbing. Deletions triggered by the handlers keep locked writes.
- manager.rs: the four *_notification_handler paths (8 call sites)
  switch to the lock-free variants.
- Integration test: while the lock quorum is unavailable (DistErasure
  with empty lockers), load_group_no_lock must succeed exactly where
  the locked load_group fails.

Request-path loads (check_key, verify_temp_user_persistence) and admin
write-then-reload paths intentionally stay locked: load_user_identity
embeds expiry deletions, so those need the side-effect extraction
tracked in rustfs/backlog#884 before going lock-free.

Verification:
- cargo test -p rustfs-iam --lib (156 passed)
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- make pre-commit

Ref: rustfs/backlog#884, rustfs#4304

Co-authored-by: heihutu <heihutu@gmail.com>

* fix(server): drop unused iam_bootstrap_failure_hint import in tests

The hint tests live in their own hint_tests module with a local import;
the stale re-import in mod tests failed clippy's -D warnings on the
Test and Lint CI variants.

Verification:
- cargo clippy -p rustfs --all-targets

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix

---------

Co-authored-by: heihutu <heihutu@gmail.com>
This commit is contained in:
houseme
2026-07-07 20:07:09 +08:00
committed by GitHub
parent 2247823200
commit 7001373316
11 changed files with 735 additions and 155 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Cargo.lock
Generated
+17 -16
View File
@@ -3603,7 +3603,7 @@ dependencies = [
"libc", "libc",
"option-ext", "option-ext",
"redox_users", "redox_users",
"windows-sys 0.59.0", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -3926,7 +3926,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [ dependencies = [
"libc", "libc",
"windows-sys 0.59.0", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -6466,7 +6466,7 @@ version = "0.50.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5" checksum = "7957b9740744892f114936ab4a57b3f487491bbeafaf8083688b16841a4240e5"
dependencies = [ dependencies = [
"windows-sys 0.59.0", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -6604,7 +6604,7 @@ version = "5.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "51e219e79014df21a225b1860a479e2dcd7cbd9130f4defd4bd0e191ea31d67d" checksum = "51e219e79014df21a225b1860a479e2dcd7cbd9130f4defd4bd0e191ea31d67d"
dependencies = [ dependencies = [
"base64 0.21.7", "base64 0.22.1",
"chrono", "chrono",
"getrandom 0.2.17", "getrandom 0.2.17",
"http 1.4.2", "http 1.4.2",
@@ -7716,7 +7716,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf" checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
dependencies = [ dependencies = [
"heck", "heck",
"itertools 0.10.5", "itertools 0.14.0",
"log", "log",
"multimap", "multimap",
"once_cell", "once_cell",
@@ -7736,7 +7736,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042" checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
dependencies = [ dependencies = [
"heck", "heck",
"itertools 0.10.5", "itertools 0.14.0",
"log", "log",
"multimap", "multimap",
"petgraph 0.8.3", "petgraph 0.8.3",
@@ -7757,7 +7757,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"itertools 0.10.5", "itertools 0.14.0",
"proc-macro2", "proc-macro2",
"quote", "quote",
"syn 2.0.118", "syn 2.0.118",
@@ -7770,7 +7770,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf" checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"itertools 0.10.5", "itertools 0.14.0",
"proc-macro2", "proc-macro2",
"quote", "quote",
"syn 2.0.118", "syn 2.0.118",
@@ -7988,7 +7988,7 @@ dependencies = [
"once_cell", "once_cell",
"socket2", "socket2",
"tracing", "tracing",
"windows-sys 0.59.0", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -9198,6 +9198,7 @@ dependencies = [
"serde_json", "serde_json",
"serial_test", "serial_test",
"temp-env", "temp-env",
"tempfile",
"thiserror 2.0.18", "thiserror 2.0.18",
"time", "time",
"tokio", "tokio",
@@ -10019,7 +10020,7 @@ dependencies = [
"errno", "errno",
"libc", "libc",
"linux-raw-sys 0.12.1", "linux-raw-sys 0.12.1",
"windows-sys 0.59.0", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -10093,7 +10094,7 @@ dependencies = [
"security-framework", "security-framework",
"security-framework-sys", "security-framework-sys",
"webpki-root-certs", "webpki-root-certs",
"windows-sys 0.59.0", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -10116,9 +10117,9 @@ dependencies = [
[[package]] [[package]]
name = "rustversion" name = "rustversion"
version = "1.0.22" version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
[[package]] [[package]]
name = "rusty-fork" name = "rusty-fork"
@@ -11254,10 +11255,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [ dependencies = [
"fastrand", "fastrand",
"getrandom 0.3.4", "getrandom 0.4.3",
"once_cell", "once_cell",
"rustix 1.1.4", "rustix 1.1.4",
"windows-sys 0.59.0", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
@@ -12338,7 +12339,7 @@ version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [ dependencies = [
"windows-sys 0.59.0", "windows-sys 0.61.2",
] ]
[[package]] [[package]]
crates/iam/Cargo.toml
+2 -1
View File
@@ -60,7 +60,8 @@ url = { workspace = true }
[dev-dependencies] [dev-dependencies]
pollster.workspace = true pollster.workspace = true
serial_test = { workspace = true } serial_test = { workspace = true }
temp-env = { workspace = true } temp-env = { workspace = true, features = ["async_closure"] }
tempfile = { workspace = true }
[lib] [lib]
doctest = false doctest = false
crates/iam/src/manager.rs
+12 -8
View File
@@ -1847,7 +1847,7 @@ where
pub async fn group_notification_handler(&self, group: &str) -> Result<()> { pub async fn group_notification_handler(&self, group: &str) -> Result<()> {
let mut m = HashMap::new(); let mut m = HashMap::new();
if let Err(err) = self.api.load_group(group, &mut m).await { if let Err(err) = self.api.load_group_no_lock(group, &mut m).await {
if !is_err_no_such_group(&err) { if !is_err_no_such_group(&err) {
return Err(err); return Err(err);
} }
@@ -1876,7 +1876,7 @@ where
pub async fn policy_notification_handler(&self, policy: &str) -> Result<()> { pub async fn policy_notification_handler(&self, policy: &str) -> Result<()> {
let mut m = HashMap::new(); let mut m = HashMap::new();
if let Err(err) = self.api.load_policy_doc(policy, &mut m).await { if let Err(err) = self.api.load_policy_doc_no_lock(policy, &mut m).await {
if !is_err_no_such_policy(&err) { if !is_err_no_such_policy(&err) {
return Err(err); return Err(err);
} }
@@ -1927,7 +1927,7 @@ where
pub async fn policy_mapping_notification_handler(&self, name: &str, user_type: UserType, is_group: bool) -> Result<()> { pub async fn policy_mapping_notification_handler(&self, name: &str, user_type: UserType, is_group: bool) -> Result<()> {
let mut m = HashMap::new(); let mut m = HashMap::new();
if let Err(err) = self.api.load_mapped_policy(name, user_type, is_group, &mut m).await { if let Err(err) = self.api.load_mapped_policy_no_lock(name, user_type, is_group, &mut m).await {
if !is_err_no_such_policy(&err) { if !is_err_no_such_policy(&err) {
return Err(err); return Err(err);
} }
@@ -1963,7 +1963,7 @@ where
} }
pub async fn user_notification_handler(&self, name: &str, user_type: UserType) -> Result<()> { pub async fn user_notification_handler(&self, name: &str, user_type: UserType) -> Result<()> {
let mut m = HashMap::new(); let mut m = HashMap::new();
if let Err(err) = self.api.load_user(name, user_type, &mut m).await { if let Err(err) = self.api.load_user_no_lock(name, user_type, &mut m).await {
if !is_err_no_such_user(&err) { if !is_err_no_such_user(&err) {
return Err(err); return Err(err);
} }
@@ -2028,7 +2028,7 @@ where
let mut policies = HashMap::new(); let mut policies = HashMap::new();
if let Err(err) = self if let Err(err) = self
.api .api
.load_mapped_policy(&parent_user, user_type, false, &mut policies) .load_mapped_policy_no_lock(&parent_user, user_type, false, &mut policies)
.await .await
{ {
if !is_err_no_such_policy(&err) { if !is_err_no_such_policy(&err) {
@@ -2040,7 +2040,11 @@ where
} }
UserType::Reg => { UserType::Reg => {
let mut policies = HashMap::new(); let mut policies = HashMap::new();
if let Err(err) = self.api.load_mapped_policy(name, user_type, false, &mut policies).await { if let Err(err) = self
.api
.load_mapped_policy_no_lock(name, user_type, false, &mut policies)
.await
{
if !is_err_no_such_policy(&err) { if !is_err_no_such_policy(&err) {
return Err(err); return Err(err);
} }
@@ -2063,7 +2067,7 @@ where
let mut policies = HashMap::new(); let mut policies = HashMap::new();
if let Err(err) = self if let Err(err) = self
.api .api
.load_mapped_policy(&parent_user, UserType::Reg, false, &mut policies) .load_mapped_policy_no_lock(&parent_user, UserType::Reg, false, &mut policies)
.await .await
{ {
if !is_err_no_such_policy(&err) { if !is_err_no_such_policy(&err) {
@@ -2076,7 +2080,7 @@ where
let mut policies = HashMap::new(); let mut policies = HashMap::new();
if let Err(err) = self if let Err(err) = self
.api .api
.load_mapped_policy(&parent_user, UserType::Sts, false, &mut policies) .load_mapped_policy_no_lock(&parent_user, UserType::Sts, false, &mut policies)
.await .await
{ {
if !is_err_no_such_policy(&err) { if !is_err_no_such_policy(&err) {
crates/iam/src/store.rs
+29
View File
@@ -71,6 +71,35 @@ pub trait Store: Clone + Send + Sync + 'static {
) -> Result<()>; ) -> Result<()>;
async fn load_all(&self, cache: &Cache) -> Result<()>; async fn load_all(&self, cache: &Cache) -> Result<()>;
// Lock-free variants used by the cross-node notification handlers.
//
// Notification-path cache refreshes are asynchronous, best-effort, and
// already tolerate stale values (the periodic reload converges them), so
// they must not depend on the node-counted namespace-lock quorum — the
// same rationale as the lock-free bootstrap `load_all` (rustfs#4304;
// MinIO's readConfig takes no distributed lock either). The defaults
// forward to the locked variants so existing `Store` implementations
// (including test mocks) keep their behavior; `ObjectStore` overrides
// them with lock-free reads.
async fn load_user_no_lock(&self, name: &str, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()> {
self.load_user(name, user_type, m).await
}
async fn load_group_no_lock(&self, name: &str, m: &mut HashMap<String, GroupInfo>) -> Result<()> {
self.load_group(name, m).await
}
async fn load_policy_doc_no_lock(&self, name: &str, m: &mut HashMap<String, PolicyDoc>) -> Result<()> {
self.load_policy_doc(name, m).await
}
async fn load_mapped_policy_no_lock(
&self,
name: &str,
user_type: UserType,
is_group: bool,
m: &mut HashMap<String, MappedPolicy>,
) -> Result<()> {
self.load_mapped_policy(name, user_type, is_group, m).await
}
} }
#[derive(Debug, PartialEq, Eq, Clone, Copy)] #[derive(Debug, PartialEq, Eq, Clone, Copy)]
crates/iam/src/store/object.rs
+272 -121
View File
@@ -95,6 +95,38 @@ fn get_mapped_policy_path(name: &str, user_type: UserType, is_group: bool) -> St
} }
} }
/// Lock semantics for IAM config reads.
///
/// Bootstrap/snapshot loads (`load_all`) must not require distributed
/// namespace locks: lock quorum is counted over cluster nodes, so during a
/// sequential restart the peers are unreachable and every locked read fails
/// with `QuorumNotReached` even though the storage read quorum is already
/// satisfiable (rustfs#4304). This mirrors the startup contract established
/// for pool metadata in rustfs#4056. Config objects are atomic whole-object
/// writes, so a lock-free read only ever observes an old or a new value;
/// staleness is bounded by the periodic IAM reload.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum LoadMode {
/// On-line load on behalf of a live request: takes namespace locks.
Locked,
/// Cache-refresh load (bootstrap/reload `load_all`, or a notification-path
/// single-object refresh): bypasses namespace locks.
BootstrapNoLock,
}
impl LoadMode {
fn no_lock(self) -> bool {
matches!(self, LoadMode::BootstrapNoLock)
}
fn read_opts(self) -> IamObjectOptions {
IamObjectOptions {
no_lock: self.no_lock(),
..Default::default()
}
}
}
#[derive(Debug)] #[derive(Debug)]
pub struct StringOrErr { pub struct StringOrErr {
pub item: Option<String>, pub item: Option<String>,
@@ -359,9 +391,13 @@ impl ObjectStore {
Self::prepare_data_for_storage(data) Self::prepare_data_for_storage(data)
} }
async fn load_iamconfig_bytes_with_metadata(&self, path: impl AsRef<str> + Send) -> Result<(Vec<u8>, IamObjectInfo)> { async fn load_iamconfig_bytes_with_metadata(
&self,
path: impl AsRef<str> + Send,
mode: LoadMode,
) -> Result<(Vec<u8>, IamObjectInfo)> {
let path_ref = path.as_ref(); let path_ref = path.as_ref();
let (data, obj) = read_iam_config_with_metadata(self.object_api.clone(), path_ref, &IamObjectOptions::default()).await?; let (data, obj) = read_iam_config_with_metadata(self.object_api.clone(), path_ref, &mode.read_opts()).await?;
let outcome = Self::decrypt_data_with_source(&data)?; let outcome = Self::decrypt_data_with_source(&data)?;
self.maybe_schedule_lazy_rewrite(path_ref, &outcome, &obj); self.maybe_schedule_lazy_rewrite(path_ref, &outcome, &obj);
@@ -472,13 +508,13 @@ impl ObjectStore {
Ok(res) Ok(res)
} }
async fn load_policy_doc_concurrent(&self, names: &[String]) -> Result<Vec<PolicyDoc>> { async fn load_policy_doc_concurrent(&self, names: &[String], mode: LoadMode) -> Result<Vec<PolicyDoc>> {
let mut futures = Vec::with_capacity(names.len()); let mut futures = Vec::with_capacity(names.len());
for name in names { for name in names {
let policy_name = rustfs_utils::path::dir(name); let policy_name = rustfs_utils::path::dir(name);
futures.push(async move { futures.push(async move {
match self.load_policy(&policy_name).await { match self.load_policy_with(&policy_name, mode).await {
Ok(p) => Ok(p), Ok(p) => Ok(p),
Err(err) => { Err(err) => {
if !is_err_no_such_policy(&err) { if !is_err_no_such_policy(&err) {
@@ -504,13 +540,13 @@ impl ObjectStore {
Ok(policies) Ok(policies)
} }
async fn load_user_concurrent(&self, names: &[String], user_type: UserType) -> Result<Vec<UserIdentity>> { async fn load_user_concurrent(&self, names: &[String], user_type: UserType, mode: LoadMode) -> Result<Vec<UserIdentity>> {
let mut futures = Vec::with_capacity(names.len()); let mut futures = Vec::with_capacity(names.len());
for name in names { for name in names {
let user_name = rustfs_utils::path::dir(name); let user_name = rustfs_utils::path::dir(name);
futures.push(async move { futures.push(async move {
match self.load_user_identity(&user_name, user_type).await { match self.load_user_identity_with(&user_name, user_type, mode).await {
Ok(res) => Ok(res), Ok(res) => Ok(res),
Err(err) => { Err(err) => {
if !is_err_no_such_user(&err) { if !is_err_no_such_user(&err) {
@@ -535,9 +571,155 @@ impl ObjectStore {
Ok(users) Ok(users)
} }
async fn load_mapped_policy_internal(&self, name: &str, user_type: UserType, is_group: bool) -> Result<MappedPolicy> { /// Parameterized core of [`Store::load_iam_config`]. `LoadMode::BootstrapNoLock`
/// keeps the bootstrap snapshot load free of namespace locks (rustfs#4304).
async fn load_iam_config_with<Item: DeserializeOwned>(&self, path: impl AsRef<str> + Send, mode: LoadMode) -> Result<Item> {
let path_ref = path.as_ref();
let (data, obj) = read_iam_config_with_metadata(self.object_api.clone(), path_ref, &mode.read_opts()).await?;
let outcome = match Self::decrypt_data_with_source(&data) {
Ok(v) => v,
Err(err) => {
warn!(path = %path_ref, error = %err, "IAM config decrypt failed; keeping file");
// keep the config file when decrypt failed - do not delete
return Err(Error::ConfigNotFound);
}
};
self.maybe_schedule_lazy_rewrite(path_ref, &outcome, &obj);
Ok(serde_json::from_slice(&outcome.plain)?)
}
/// Parameterized core of [`Store::load_user_identity`].
async fn load_user_identity_with(&self, name: &str, user_type: UserType, mode: LoadMode) -> Result<UserIdentity> {
let mut u: UserIdentity = self
.load_iam_config_with(get_user_identity_path(name, user_type), mode)
.await
.map_err(|err| {
if is_err_config_not_found(&err) {
warn!(name, user_type = ?user_type, "IAM user identity missing");
Error::NoSuchUser(name.to_owned())
} else {
warn!(name, user_type = ?user_type, error = ?err, "IAM user identity load failed");
err
}
})?;
if u.credentials.is_expired() {
let _ = self.delete_iam_config(get_user_identity_path(name, user_type)).await;
let _ = self.delete_iam_config(get_mapped_policy_path(name, user_type, false)).await;
warn!(name, user_type = ?user_type, "IAM user identity expired and was removed");
return Err(Error::NoSuchUser(name.to_owned()));
}
if u.credentials.access_key.is_empty() {
u.credentials.access_key = name.to_owned();
}
if !u.credentials.session_token.is_empty() {
let claims_result = if user_type == UserType::Svc && u.credentials.expiration.is_none() {
extract_jwt_claims_allow_missing_exp(&u)
} else {
extract_jwt_claims(&u)
};
match claims_result {
Ok(claims) => {
u.credentials.claims = Some(claims);
}
Err(err) => {
if u.credentials.is_temp() {
let _ = self.delete_iam_config(get_user_identity_path(name, user_type)).await;
let _ = self.delete_iam_config(get_mapped_policy_path(name, user_type, false)).await;
}
warn!(name, user_type = ?user_type, error = ?err, "IAM JWT claim extraction failed");
return Err(Error::NoSuchUser(name.to_owned()));
}
}
}
Ok(u)
}
/// Parameterized core of [`Store::load_user`].
async fn load_user_with(
&self,
name: &str,
user_type: UserType,
m: &mut HashMap<String, UserIdentity>,
mode: LoadMode,
) -> Result<()> {
self.load_user_identity_with(name, user_type, mode).await.map(|u| {
m.insert(name.to_owned(), u);
})
}
/// Parameterized core of [`Store::load_group`].
async fn load_group_with(&self, name: &str, m: &mut HashMap<String, GroupInfo>, mode: LoadMode) -> Result<()> {
let u: GroupInfo = self
.load_iam_config_with(get_group_info_path(name), mode)
.await
.map_err(|err| {
if is_err_config_not_found(&err) {
Error::NoSuchGroup(name.to_string())
} else {
err
}
})?;
m.insert(name.to_owned(), u);
Ok(())
}
/// Parameterized core of [`Store::load_policy`].
async fn load_policy_with(&self, name: &str, mode: LoadMode) -> Result<PolicyDoc> {
let (data, obj) = self
.load_iamconfig_bytes_with_metadata(get_policy_doc_path(name), mode)
.await
.map_err(|err| {
if is_err_config_not_found(&err) {
Error::NoSuchPolicy
} else {
err
}
})?;
let mut info = PolicyDoc::try_from(data)?;
if info.version == 0 {
info.create_date = obj.mod_time;
info.update_date = obj.mod_time;
}
Ok(info)
}
/// Parameterized core of [`Store::load_mapped_policy`].
async fn load_mapped_policy_with(
&self,
name: &str,
user_type: UserType,
is_group: bool,
m: &mut HashMap<String, MappedPolicy>,
mode: LoadMode,
) -> Result<()> {
let info = self.load_mapped_policy_internal(name, user_type, is_group, mode).await?;
m.insert(name.to_owned(), info);
Ok(())
}
async fn load_mapped_policy_internal(
&self,
name: &str,
user_type: UserType,
is_group: bool,
mode: LoadMode,
) -> Result<MappedPolicy> {
let info: MappedPolicy = self let info: MappedPolicy = self
.load_iam_config(get_mapped_policy_path(name, user_type, is_group)) .load_iam_config_with(get_mapped_policy_path(name, user_type, is_group), mode)
.await .await
.map_err(|err| { .map_err(|err| {
if is_err_config_not_found(&err) { if is_err_config_not_found(&err) {
@@ -555,13 +737,14 @@ impl ObjectStore {
names: &[String], names: &[String],
user_type: UserType, user_type: UserType,
is_group: bool, is_group: bool,
mode: LoadMode,
) -> Result<Vec<MappedPolicy>> { ) -> Result<Vec<MappedPolicy>> {
let mut futures = Vec::with_capacity(names.len()); let mut futures = Vec::with_capacity(names.len());
for name in names { for name in names {
let policy_name = name.trim_end_matches(".json"); let policy_name = name.trim_end_matches(".json");
futures.push(async move { futures.push(async move {
match self.load_mapped_policy_internal(policy_name, user_type, is_group).await { match self.load_mapped_policy_internal(policy_name, user_type, is_group, mode).await {
Ok(p) => Ok(p), Ok(p) => Ok(p),
Err(err) => { Err(err) => {
if !is_err_no_such_policy(&err) { if !is_err_no_such_policy(&err) {
@@ -615,21 +798,7 @@ impl Store for ObjectStore {
false false
} }
async fn load_iam_config<Item: DeserializeOwned>(&self, path: impl AsRef<str> + Send) -> Result<Item> { async fn load_iam_config<Item: DeserializeOwned>(&self, path: impl AsRef<str> + Send) -> Result<Item> {
let path_ref = path.as_ref(); self.load_iam_config_with(path, LoadMode::Locked).await
let (data, obj) = read_iam_config_with_metadata(self.object_api.clone(), path_ref, &IamObjectOptions::default()).await?;
let outcome = match Self::decrypt_data_with_source(&data) {
Ok(v) => v,
Err(err) => {
warn!(path = %path_ref, error = %err, "IAM config decrypt failed; keeping file");
// keep the config file when decrypt failed - do not delete
return Err(Error::ConfigNotFound);
}
};
self.maybe_schedule_lazy_rewrite(path_ref, &outcome, &obj);
Ok(serde_json::from_slice(&outcome.plain)?)
} }
/// Saves IAM configuration with a retry mechanism on failure. /// Saves IAM configuration with a retry mechanism on failure.
/// ///
@@ -712,58 +881,10 @@ impl Store for ObjectStore {
Ok(()) Ok(())
} }
async fn load_user_identity(&self, name: &str, user_type: UserType) -> Result<UserIdentity> { async fn load_user_identity(&self, name: &str, user_type: UserType) -> Result<UserIdentity> {
let mut u: UserIdentity = self self.load_user_identity_with(name, user_type, LoadMode::Locked).await
.load_iam_config(get_user_identity_path(name, user_type))
.await
.map_err(|err| {
if is_err_config_not_found(&err) {
warn!(name, user_type = ?user_type, "IAM user identity missing");
Error::NoSuchUser(name.to_owned())
} else {
warn!(name, user_type = ?user_type, error = ?err, "IAM user identity load failed");
err
}
})?;
if u.credentials.is_expired() {
let _ = self.delete_iam_config(get_user_identity_path(name, user_type)).await;
let _ = self.delete_iam_config(get_mapped_policy_path(name, user_type, false)).await;
warn!(name, user_type = ?user_type, "IAM user identity expired and was removed");
return Err(Error::NoSuchUser(name.to_owned()));
}
if u.credentials.access_key.is_empty() {
u.credentials.access_key = name.to_owned();
}
if !u.credentials.session_token.is_empty() {
let claims_result = if user_type == UserType::Svc && u.credentials.expiration.is_none() {
extract_jwt_claims_allow_missing_exp(&u)
} else {
extract_jwt_claims(&u)
};
match claims_result {
Ok(claims) => {
u.credentials.claims = Some(claims);
}
Err(err) => {
if u.credentials.is_temp() {
let _ = self.delete_iam_config(get_user_identity_path(name, user_type)).await;
let _ = self.delete_iam_config(get_mapped_policy_path(name, user_type, false)).await;
}
warn!(name, user_type = ?user_type, error = ?err, "IAM JWT claim extraction failed");
return Err(Error::NoSuchUser(name.to_owned()));
}
}
}
Ok(u)
} }
async fn load_user(&self, name: &str, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()> { async fn load_user(&self, name: &str, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()> {
self.load_user_identity(name, user_type).await.map(|u| { self.load_user_with(name, user_type, m, LoadMode::Locked).await
m.insert(name.to_owned(), u);
})
} }
async fn load_users(&self, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()> { async fn load_users(&self, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()> {
let base_prefix = match user_type { let base_prefix = match user_type {
@@ -823,16 +944,7 @@ impl Store for ObjectStore {
Ok(()) Ok(())
} }
async fn load_group(&self, name: &str, m: &mut HashMap<String, GroupInfo>) -> Result<()> { async fn load_group(&self, name: &str, m: &mut HashMap<String, GroupInfo>) -> Result<()> {
let u: GroupInfo = self.load_iam_config(get_group_info_path(name)).await.map_err(|err| { self.load_group_with(name, m, LoadMode::Locked).await
if is_err_config_not_found(&err) {
Error::NoSuchGroup(name.to_string())
} else {
err
}
})?;
m.insert(name.to_owned(), u);
Ok(())
} }
async fn load_groups(&self, m: &mut HashMap<String, GroupInfo>) -> Result<()> { async fn load_groups(&self, m: &mut HashMap<String, GroupInfo>) -> Result<()> {
let ctx = CancellationToken::new(); let ctx = CancellationToken::new();
@@ -875,25 +987,7 @@ impl Store for ObjectStore {
Ok(()) Ok(())
} }
async fn load_policy(&self, name: &str) -> Result<PolicyDoc> { async fn load_policy(&self, name: &str) -> Result<PolicyDoc> {
let (data, obj) = self self.load_policy_with(name, LoadMode::Locked).await
.load_iamconfig_bytes_with_metadata(get_policy_doc_path(name))
.await
.map_err(|err| {
if is_err_config_not_found(&err) {
Error::NoSuchPolicy
} else {
err
}
})?;
let mut info = PolicyDoc::try_from(data)?;
if info.version == 0 {
info.create_date = obj.mod_time;
info.update_date = obj.mod_time;
}
Ok(info)
} }
async fn load_policy_doc(&self, name: &str, m: &mut HashMap<String, PolicyDoc>) -> Result<()> { async fn load_policy_doc(&self, name: &str, m: &mut HashMap<String, PolicyDoc>) -> Result<()> {
@@ -955,11 +1049,8 @@ impl Store for ObjectStore {
is_group: bool, is_group: bool,
m: &mut HashMap<String, MappedPolicy>, m: &mut HashMap<String, MappedPolicy>,
) -> Result<()> { ) -> Result<()> {
let info = self.load_mapped_policy_internal(name, user_type, is_group).await?; self.load_mapped_policy_with(name, user_type, is_group, m, LoadMode::Locked)
.await
m.insert(name.to_owned(), info);
Ok(())
} }
async fn load_mapped_policies( async fn load_mapped_policies(
&self, &self,
@@ -1000,7 +1091,39 @@ impl Store for ObjectStore {
Ok(()) Ok(())
} }
// Notification-path cache refreshes read lock-free (see the trait docs):
// they are best-effort and stale-tolerant, so they must not fail on the
// node-counted lock quorum. Deletions triggered by these refreshes keep
// their locked write semantics.
async fn load_user_no_lock(&self, name: &str, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()> {
self.load_user_with(name, user_type, m, LoadMode::BootstrapNoLock).await
}
async fn load_group_no_lock(&self, name: &str, m: &mut HashMap<String, GroupInfo>) -> Result<()> {
self.load_group_with(name, m, LoadMode::BootstrapNoLock).await
}
async fn load_policy_doc_no_lock(&self, name: &str, m: &mut HashMap<String, PolicyDoc>) -> Result<()> {
let info = self.load_policy_with(name, LoadMode::BootstrapNoLock).await?;
m.insert(name.to_owned(), info);
Ok(())
}
async fn load_mapped_policy_no_lock(
&self,
name: &str,
user_type: UserType,
is_group: bool,
m: &mut HashMap<String, MappedPolicy>,
) -> Result<()> {
self.load_mapped_policy_with(name, user_type, is_group, m, LoadMode::BootstrapNoLock)
.await
}
async fn load_all(&self, cache: &Cache) -> Result<()> { async fn load_all(&self, cache: &Cache) -> Result<()> {
// Bulk snapshot load: must stay free of namespace locks so IAM
// bootstrap only depends on the storage read quorum, never on the
// node-counted lock quorum (rustfs#4304; startup contract rustfs#4056).
const LOAD_ALL_MODE: LoadMode = LoadMode::BootstrapNoLock;
let cache_snapshot = cache.snapshot(); let cache_snapshot = cache.snapshot();
let listed_config_items = self.list_all_iamconfig_items().await?; let listed_config_items = self.list_all_iamconfig_items().await?;
@@ -1011,7 +1134,7 @@ impl Store for ObjectStore {
loop { loop {
if policies_list.len() < 32 { if policies_list.len() < 32 {
let policy_docs = self.load_policy_doc_concurrent(&policies_list).await?; let policy_docs = self.load_policy_doc_concurrent(&policies_list, LOAD_ALL_MODE).await?;
for (idx, p) in policy_docs.into_iter().enumerate() { for (idx, p) in policy_docs.into_iter().enumerate() {
if p.policy.version.is_empty() { if p.policy.version.is_empty() {
@@ -1027,7 +1150,7 @@ impl Store for ObjectStore {
break; break;
} }
let policy_docs = self.load_policy_doc_concurrent(&policies_list).await?; let policy_docs = self.load_policy_doc_concurrent(&policies_list, LOAD_ALL_MODE).await?;
for (idx, p) in policy_docs.into_iter().enumerate() { for (idx, p) in policy_docs.into_iter().enumerate() {
if p.policy.version.is_empty() { if p.policy.version.is_empty() {
@@ -1051,7 +1174,9 @@ impl Store for ObjectStore {
loop { loop {
if item_name_list.len() < 32 { if item_name_list.len() < 32 {
let items = self.load_user_concurrent(&item_name_list, UserType::Reg).await?; let items = self
.load_user_concurrent(&item_name_list, UserType::Reg, LOAD_ALL_MODE)
.await?;
for (idx, p) in items.into_iter().enumerate() { for (idx, p) in items.into_iter().enumerate() {
if p.credentials.access_key.is_empty() { if p.credentials.access_key.is_empty() {
@@ -1065,7 +1190,9 @@ impl Store for ObjectStore {
break; break;
} }
let items = self.load_user_concurrent(&item_name_list, UserType::Reg).await?; let items = self
.load_user_concurrent(&item_name_list, UserType::Reg, LOAD_ALL_MODE)
.await?;
for (idx, p) in items.into_iter().enumerate() { for (idx, p) in items.into_iter().enumerate() {
if p.credentials.access_key.is_empty() { if p.credentials.access_key.is_empty() {
@@ -1089,7 +1216,7 @@ impl Store for ObjectStore {
for item in item_name_list.iter() { for item in item_name_list.iter() {
let name = rustfs_utils::path::dir(item); let name = rustfs_utils::path::dir(item);
debug!(group = %name, "IAM group loaded"); debug!(group = %name, "IAM group loaded");
if let Err(err) = self.load_group(&name, &mut items_cache).await { if let Err(err) = self.load_group_with(&name, &mut items_cache, LOAD_ALL_MODE).await {
return Err(Error::other(format!("load group failed: {err}"))); return Err(Error::other(format!("load group failed: {err}")));
}; };
} }
@@ -1107,7 +1234,7 @@ impl Store for ObjectStore {
loop { loop {
if item_name_list.len() < 32 { if item_name_list.len() < 32 {
let items = self let items = self
.load_mapped_policy_concurrent(&item_name_list, UserType::Reg, false) .load_mapped_policy_concurrent(&item_name_list, UserType::Reg, false, LOAD_ALL_MODE)
.await?; .await?;
for (idx, p) in items.into_iter().enumerate() { for (idx, p) in items.into_iter().enumerate() {
@@ -1123,7 +1250,7 @@ impl Store for ObjectStore {
} }
let items = self let items = self
.load_mapped_policy_concurrent(&item_name_list, UserType::Reg, false) .load_mapped_policy_concurrent(&item_name_list, UserType::Reg, false, LOAD_ALL_MODE)
.await?; .await?;
for (idx, p) in items.into_iter().enumerate() { for (idx, p) in items.into_iter().enumerate() {
@@ -1151,7 +1278,9 @@ impl Store for ObjectStore {
let name = item.trim_end_matches(".json"); let name = item.trim_end_matches(".json");
debug!(group = %name, "IAM group policy loaded"); debug!(group = %name, "IAM group policy loaded");
if let Err(err) = self.load_mapped_policy(name, UserType::Reg, true, &mut items_cache).await if let Err(err) = self
.load_mapped_policy_with(name, UserType::Reg, true, &mut items_cache, LOAD_ALL_MODE)
.await
&& !is_err_no_such_policy(&err) && !is_err_no_such_policy(&err)
{ {
return Err(Error::other(format!("load group policy failed: {err}"))); return Err(Error::other(format!("load group policy failed: {err}")));
@@ -1170,7 +1299,9 @@ impl Store for ObjectStore {
for item in item_name_list.iter() { for item in item_name_list.iter() {
let name = rustfs_utils::path::dir(item); let name = rustfs_utils::path::dir(item);
debug!(user = %name, "IAM service user loaded"); debug!(user = %name, "IAM service user loaded");
if let Err(err) = self.load_user(&name, UserType::Svc, &mut items_cache).await if let Err(err) = self
.load_user_with(&name, UserType::Svc, &mut items_cache, LOAD_ALL_MODE)
.await
&& !is_err_no_such_user(&err) && !is_err_no_such_user(&err)
{ {
return Err(Error::other(format!("load svc user failed: {err}"))); return Err(Error::other(format!("load svc user failed: {err}")));
@@ -1182,7 +1313,7 @@ impl Store for ObjectStore {
if !user_items_cache.contains_key(&parent) { if !user_items_cache.contains_key(&parent) {
debug!(user = %parent, "IAM STS parent policy loaded"); debug!(user = %parent, "IAM STS parent policy loaded");
if let Err(err) = self if let Err(err) = self
.load_mapped_policy(&parent, UserType::Sts, false, &mut sts_policies_cache) .load_mapped_policy_with(&parent, UserType::Sts, false, &mut sts_policies_cache, LOAD_ALL_MODE)
.await .await
&& !is_err_no_such_policy(&err) && !is_err_no_such_policy(&err)
{ {
@@ -1203,7 +1334,10 @@ impl Store for ObjectStore {
let name = rustfs_utils::path::dir(item); let name = rustfs_utils::path::dir(item);
debug!(user = %name, "IAM STS user loaded"); debug!(user = %name, "IAM STS user loaded");
if let Err(err) = self.load_user(&name, UserType::Sts, &mut sts_items_cache).await { if let Err(err) = self
.load_user_with(&name, UserType::Sts, &mut sts_items_cache, LOAD_ALL_MODE)
.await
{
debug!(user = %name, error = %err, "IAM STS user load failed"); debug!(user = %name, error = %err, "IAM STS user load failed");
}; };
} }
@@ -1215,7 +1349,7 @@ impl Store for ObjectStore {
let name = item.trim_end_matches(".json"); let name = item.trim_end_matches(".json");
debug!(user = %name, "IAM STS user policy loaded"); debug!(user = %name, "IAM STS user policy loaded");
if let Err(err) = self if let Err(err) = self
.load_mapped_policy(name, UserType::Sts, false, &mut sts_policies_cache) .load_mapped_policy_with(name, UserType::Sts, false, &mut sts_policies_cache, LOAD_ALL_MODE)
.await .await
{ {
debug!(user = %name, error = %err, "IAM STS user policy load failed"); debug!(user = %name, error = %err, "IAM STS user policy load failed");
@@ -1250,12 +1384,29 @@ impl Store for ObjectStore {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::{DecryptSource, ObjectStore}; use super::{DecryptSource, LoadMode, ObjectStore};
use crate::keyring; use crate::keyring;
use rustfs_credentials::{Credentials, init_global_action_credentials}; use rustfs_credentials::{Credentials, init_global_action_credentials};
use serial_test::serial; use serial_test::serial;
use temp_env::with_vars; use temp_env::with_vars;
/// rustfs#4304 / startup contract rustfs#4056: the bulk snapshot load
/// (`load_all`) must never depend on the node-counted namespace lock
/// quorum, so its reads have to carry `no_lock = true` down to the
/// storage layer.
#[test]
fn test_bootstrap_load_mode_bypasses_namespace_locks() {
assert!(LoadMode::BootstrapNoLock.no_lock());
assert!(LoadMode::BootstrapNoLock.read_opts().no_lock);
}
/// On-line single-object loads keep the locked read semantics.
#[test]
fn test_locked_load_mode_keeps_namespace_locks() {
assert!(!LoadMode::Locked.no_lock());
assert!(!LoadMode::Locked.read_opts().no_lock);
}
fn test_cred() -> Credentials { fn test_cred() -> Credentials {
if let Some(cred) = crate::root_credentials::credentials() { if let Some(cred) = crate::root_credentials::credentials() {
return cred; return cred;
crates/iam/tests/ecstore_test_compat/mod.rs
+35
View File
@@ -0,0 +1,35 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Test-only ECStore compatibility boundary for the IAM integration tests.
//!
//! All direct `rustfs_ecstore` facade imports used by tests in this crate
//! must go through this module (architecture migration rule:
//! `check_architecture_migration_rules.sh`). Keep the surface minimal —
//! only what the tests actually need to build a temp-disk ECStore fixture
//! and to flip the erasure setup type for lock-quorum fault injection.
#[allow(unused_imports)]
pub(crate) mod fixture {
pub(crate) use rustfs_ecstore::api::disk::endpoint::Endpoint;
pub(crate) use rustfs_ecstore::api::layout::{EndpointServerPools, Endpoints, PoolEndpoints, SetupType};
pub(crate) use rustfs_ecstore::api::storage::{ECStore, init_local_disks};
// `update_erasure_type` is a write-side global facade entry. Its use is
// restricted to reviewed storage_api boundaries; this test-only module is
// registered in check_architecture_migration_rules.sh precisely so the
// sequential-restart regression test can flip into distributed-erasure
// mode to make namespace-lock acquisition fail (rustfs#4304).
pub(crate) use rustfs_ecstore::api::global::update_erasure_type;
}
crates/iam/tests/iam_bootstrap_no_lock_test.rs
+159
View File
@@ -0,0 +1,159 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Regression test for rustfs#4304: IAM bootstrap must not depend on the
//! distributed namespace-lock quorum.
//!
//! During a sequential cluster restart the peer lock RPC endpoints are
//! unreachable, so every namespace-locked read fails with
//! `QuorumNotReached` even though the storage read quorum is already
//! satisfiable. The bulk snapshot load (`Store::load_all`) therefore has to
//! read with `no_lock = true` (startup contract rustfs#4056).
//!
//! The test builds a real 4-disk ECStore over temp dirs, seeds IAM data in
//! single-node mode, then flips the runtime into distributed-erasure mode.
//! In that mode `SetDisks::new_ns_lock` builds a distributed lock over the
//! set's lock clients — which are empty for a locally-built store — so every
//! locked read fails exactly like the sequential-restart scenario, while
//! plain storage reads keep working. The old (locked) `load_group` path must
//! fail and the lock-free `load_all` path must succeed.
mod ecstore_test_compat;
use ecstore_test_compat::fixture::{
ECStore, Endpoint, EndpointServerPools, Endpoints, PoolEndpoints, SetupType, init_local_disks, update_erasure_type,
};
use rustfs_iam::cache::Cache;
use rustfs_iam::store::object::ObjectStore;
use rustfs_iam::store::{GroupInfo, Store};
use serial_test::serial;
use std::collections::HashMap;
use std::sync::Arc;
use tokio_util::sync::CancellationToken;
const TEST_GROUP: &str = "seq-restart-group";
const TEST_MEMBERS: [&str; 2] = ["alice", "bob"];
async fn build_local_ecstore(temp_dir: &std::path::Path) -> Arc<ECStore> {
let disk_paths: Vec<_> = (1..=4).map(|i| temp_dir.join(format!("disk{i}"))).collect();
for disk_path in &disk_paths {
tokio::fs::create_dir_all(disk_path).await.unwrap();
}
let mut endpoints = Vec::new();
for (i, disk_path) in disk_paths.iter().enumerate() {
let mut endpoint = Endpoint::try_from(disk_path.to_str().unwrap()).unwrap();
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(i);
endpoints.push(endpoint);
}
let pool_endpoints = PoolEndpoints {
legacy: false,
set_count: 1,
drives_per_set: 4,
endpoints: Endpoints::from(endpoints),
cmd_line: "test".to_string(),
platform: format!("OS: {} | Arch: {}", std::env::consts::OS, std::env::consts::ARCH),
};
let endpoint_pools = EndpointServerPools::from(vec![pool_endpoints]);
init_local_disks(endpoint_pools.clone()).await.unwrap();
// Port 0 keeps this integration binary parallel-safe alongside other
// ECStore-backed tests.
let server_addr: std::net::SocketAddr = "127.0.0.1:0".parse().unwrap();
ECStore::new(server_addr, endpoint_pools, CancellationToken::new())
.await
.unwrap()
}
/// Restores single-node erasure mode even when an assertion panics, so a
/// failing run cannot poison later `#[serial]` tests in this process.
struct ErasureModeGuard;
impl Drop for ErasureModeGuard {
fn drop(&mut self) {
pollster::block_on(update_erasure_type(SetupType::Erasure));
}
}
#[tokio::test(flavor = "multi_thread")]
#[serial]
async fn load_all_bypasses_namespace_lock_quorum() {
// The lock acquire timeout is latched into a OnceLock on first use, so it
// must be shortened before the first locked operation of this process.
temp_env::async_with_vars([(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("1"))], async {
let temp_dir = tempfile::TempDir::with_prefix("rustfs_iam_no_lock_test_").unwrap();
let ecstore = build_local_ecstore(temp_dir.path()).await;
let store = ObjectStore::new(ecstore);
// Seed IAM data while namespace locks still work (single-node mode).
store
.save_group_info(TEST_GROUP, GroupInfo::new(TEST_MEMBERS.iter().map(|m| m.to_string()).collect()))
.await
.expect("seeding group info in single-node mode must succeed");
let mut baseline = HashMap::new();
store
.load_group(TEST_GROUP, &mut baseline)
.await
.expect("locked load_group must succeed in single-node mode");
assert_eq!(baseline[TEST_GROUP].members, TEST_MEMBERS, "seeded group must round-trip");
// Flip into distributed-erasure mode: new_ns_lock now builds a
// distributed lock over the set's (empty) lock-client list, so every
// locked read fails — the sequential-restart failure mode of
// rustfs#4304 (lock quorum unavailable, storage quorum healthy).
update_erasure_type(SetupType::DistErasure).await;
let _mode_guard = ErasureModeGuard;
let mut locked_read = HashMap::new();
let locked_err = store
.load_group(TEST_GROUP, &mut locked_read)
.await
.expect_err("locked load_group must fail while the lock quorum is unavailable");
assert!(locked_read.is_empty(), "failed locked read must not populate results");
// The P0 fix: the bulk snapshot load reads with no_lock = true, so it
// must succeed in exactly the state where the locked path fails.
let cache = Cache::default();
store
.load_all(&cache)
.await
.unwrap_or_else(|err| panic!("load_all must bypass namespace locks (locked path failed with: {locked_err}): {err}"));
// P3 step 1: notification-path cache refreshes use the same lock-free
// reads, so a cross-node notification must also be able to refresh
// this group while the lock quorum is unavailable.
let mut notification_read = HashMap::new();
store
.load_group_no_lock(TEST_GROUP, &mut notification_read)
.await
.expect("lock-free notification-path load_group must succeed while the lock quorum is unavailable");
assert_eq!(notification_read[TEST_GROUP].members, TEST_MEMBERS);
// Back in single-node mode the locked path works again; verify the
// data survived the whole exercise intact (fail-closed integrity).
drop(_mode_guard);
let mut recovered = HashMap::new();
store
.load_group(TEST_GROUP, &mut recovered)
.await
.expect("locked load_group must succeed again after restoring single-node mode");
assert_eq!(recovered[TEST_GROUP].members, TEST_MEMBERS, "group data must be intact");
})
.await;
}
docs/operations/rolling-restart.md
+114
View File
@@ -0,0 +1,114 @@
# Restarting a multi-node RustFS cluster
How to restart nodes of an erasure-coded multi-node cluster without losing
availability, what to expect when several nodes are down at once (sequential
cold start), and how to read the degraded-mode signals. Written for the
failure pattern reported in rustfs/rustfs#4304.
> Upgrading the binary or container image never changes the on-disk data
> format. Replacing the executable and restarting is safe; no migration step
> runs on startup.
## TL;DR
- **Rolling restart (no downtime):** restart **one node at a time**, and wait
for the restarted node to report `200` on `/health/ready` before touching
the next one. The remaining nodes keep serving traffic.
- **Sequential cold start (several nodes down):** nodes started before the
cluster has quorum come up in **degraded mode** — the process stays alive,
answers `503` with the blocking reason, and recovers **automatically** as
soon as enough peers are online. Do not restart-loop them; just keep
starting the remaining nodes.
## Why a single node cannot serve alone
Erasure coding shards every object (including internal metadata such as IAM
users, groups, and policies under `.rustfs.sys`) across the drives of a set.
Reading an object back needs a **read quorum** of shards online. With the
drives of one set spread over several nodes, one node alone can never satisfy
the read quorum — this is a mathematical property of erasure coding, not a
bug. The cluster becomes readable once enough nodes are up (for internal
configuration objects, which are written with maximum parity, that is
typically about half the nodes of a set).
Distributed locking similarly needs a majority of nodes' lock RPC endpoints.
The startup path no longer takes namespace locks while loading IAM
(rustfs/rustfs#4363), so IAM recovery depends only on the storage read
quorum.
## Rolling restart procedure
For each node, in any order, **one at a time**:
1. Restart the node (upgrade the binary/image first if this is an upgrade).
2. Wait until the node reports ready:
```bash
curl -fsS http://<node>:9000/health/ready
```
A ready node returns `200` with `"ready": true` in the JSON body.
3. Only then move on to the next node.
While one node is down, the rest of the cluster keeps quorum and serves all
traffic. If you take a second node down before the first is back, some
erasure sets may lose write or even read quorum and requests start failing —
this is the situation to avoid.
## Sequential cold start (multiple nodes down)
When the whole cluster (or several nodes) went down — power loss, host
maintenance, crash-looping deployment — and nodes are brought back one at a
time:
1. **Early nodes come up degraded.** The process does not exit. S3 requests
receive `503 Service Unavailable` with a `Retry-After: 5` header, an
`x-rustfs-readiness-pending` header, and a body naming the blocking
dependency:
- `storage_quorum` — waiting for enough nodes/disks for the erasure read
quorum;
- `iam` — storage is up, IAM cache is still loading;
- `startup_finalization` — last startup steps are being published.
2. **Logs say what the node waits for.** The IAM recovery loop retries with
backoff and logs `event="iam_bootstrap_retry_failed"` with an actionable
`hint` field (for example, "storage read quorum not met yet; waiting for
enough cluster nodes/disks to come online"). After repeated failures the
log level escalates from WARN to ERROR — this still does not kill the
process.
3. **Recovery is automatic.** As soon as enough peers are online for the
storage read quorum, the pending nodes finish IAM bootstrap on the next
retry and flip `/health/ready` to `200` on their own. No manual restart is
needed, and restarting them does not speed anything up.
4. **Check readiness detail while waiting.** `/health/ready` (and
`/minio/health/ready`) return per-dependency detail during degradation:
```bash
curl -s http://<node>:9000/health/ready | jq
```
The `details` object shows `storage` / `iam` / `lock` readiness, and
`degradedReasons` lists machine-readable causes such as
`storage_quorum_unavailable` or `lock_quorum_unavailable`.
## Tuning
- `RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS` (default `120`): how long startup
waits for full readiness before continuing in degraded mode with background
recovery. Raising it delays the listener during genuinely slow starts;
lowering it surfaces degraded mode sooner. Recovery retries continue
regardless of this limit.
## What is *not* normal
- A node process **exiting** with a fatal IAM/lock error during startup —
that fatal path was removed after v1.0.0-beta.5 (rustfs/rustfs#4304);
upgrade if you still see it.
- A node stuck degraded **after** the whole cluster is back: check network
reachability between nodes (peer RPC ports) and per-node clocks, then
inspect `degradedReasons` and the `hint` field of the IAM retry logs.
- A node shown offline in the console with no log output — tracked
separately, see rustfs/backlog#888.
rustfs/src/server/readiness.rs
+38 -5
View File
@@ -194,8 +194,22 @@ fn readiness_gate_blocks_path(path: &str, readiness: &GlobalReadiness) -> bool {
type BoxError = Box<dyn std::error::Error + Send + Sync>; type BoxError = Box<dyn std::error::Error + Send + Sync>;
type BoxBody = http_body_util::combinators::UnsyncBoxBody<Bytes, BoxError>; type BoxBody = http_body_util::combinators::UnsyncBoxBody<Bytes, BoxError>;
fn service_not_ready_response() -> Response<BoxBody> { /// Header exposing which startup dependency the readiness gate is waiting
let body: BoxBody = Full::new(Bytes::from_static(b"Service not ready")) /// on, so operators can diagnose a 503 without shell access (rustfs#4304).
const READINESS_PENDING_HEADER: &str = "x-rustfs-readiness-pending";
/// Maps the current startup stage to the dependency the gate is waiting on.
fn readiness_pending_dependency(stage: rustfs_common::SystemStage) -> &'static str {
match stage {
rustfs_common::SystemStage::Booting => "storage_quorum",
rustfs_common::SystemStage::StorageReady => "iam",
rustfs_common::SystemStage::IamReady | rustfs_common::SystemStage::FullReady => "startup_finalization",
}
}
fn service_not_ready_response(stage: rustfs_common::SystemStage) -> Response<BoxBody> {
let pending = readiness_pending_dependency(stage);
let body: BoxBody = Full::new(Bytes::from(format!("Service not ready: waiting for {pending}")))
.map_err(|e| -> BoxError { Box::new(e) }) .map_err(|e| -> BoxError { Box::new(e) })
.boxed_unsync(); .boxed_unsync();
@@ -211,6 +225,9 @@ fn service_not_ready_response() -> Response<BoxBody> {
.headers_mut() .headers_mut()
.insert(http::header::CACHE_CONTROL, HeaderValue::from_static("no-store")); .insert(http::header::CACHE_CONTROL, HeaderValue::from_static("no-store"));
response response
.headers_mut()
.insert(READINESS_PENDING_HEADER, HeaderValue::from_static(pending));
response
} }
impl<S, B> Service<HttpRequest<Incoming>> for ReadinessGateService<S> impl<S, B> Service<HttpRequest<Incoming>> for ReadinessGateService<S>
@@ -236,7 +253,7 @@ where
let path = req.uri().path(); let path = req.uri().path();
debug!("ReadinessGateService: Received request for path: {}", path); debug!("ReadinessGateService: Received request for path: {}", path);
if readiness_gate_blocks_path(path, &readiness) { if readiness_gate_blocks_path(path, &readiness) {
return Ok(service_not_ready_response()); return Ok(service_not_ready_response(readiness.current_stage()));
} }
let resp = inner.call(req).await?; let resp = inner.call(req).await?;
// System is ready, forward to the actual S3/RPC handlers // System is ready, forward to the actual S3/RPC handlers
@@ -1205,7 +1222,7 @@ mod tests {
#[tokio::test] #[tokio::test]
async fn service_not_ready_response_preserves_observable_contract() { async fn service_not_ready_response_preserves_observable_contract() {
let response = service_not_ready_response(); let response = service_not_ready_response(rustfs_common::SystemStage::Booting);
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE); assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!( assert_eq!(
@@ -1229,11 +1246,27 @@ mod tests {
.and_then(|value| value.to_str().ok()), .and_then(|value| value.to_str().ok()),
Some("text/plain; charset=utf-8") Some("text/plain; charset=utf-8")
); );
assert_eq!(
response
.headers()
.get(READINESS_PENDING_HEADER)
.and_then(|value| value.to_str().ok()),
Some("storage_quorum")
);
let body = match response.into_body().collect().await { let body = match response.into_body().collect().await {
Ok(body) => body.to_bytes(), Ok(body) => body.to_bytes(),
Err(err) => panic!("not-ready body should collect: {err}"), Err(err) => panic!("not-ready body should collect: {err}"),
}; };
assert_eq!(body, Bytes::from_static(b"Service not ready")); assert_eq!(body, Bytes::from_static(b"Service not ready: waiting for storage_quorum"));
}
/// rustfs#4304: operators must be able to tell from the 503 alone which
/// startup dependency the node is blocked on.
#[test]
fn readiness_pending_dependency_maps_every_stage() {
assert_eq!(readiness_pending_dependency(rustfs_common::SystemStage::Booting), "storage_quorum");
assert_eq!(readiness_pending_dependency(rustfs_common::SystemStage::StorageReady), "iam");
assert_eq!(readiness_pending_dependency(rustfs_common::SystemStage::IamReady), "startup_finalization");
} }
#[test] #[test]
rustfs/src/startup_iam.rs
+52 -4
View File
@@ -168,6 +168,7 @@ async fn run_iam_recovery_loop<InitFn, FinalizeFn>(
} }
Err(err) => { Err(err) => {
let next_interval = compute_backoff_interval(attempts + 1, initial_interval, max_interval); let next_interval = compute_backoff_interval(attempts + 1, initial_interval, max_interval);
let hint = iam_bootstrap_failure_hint(&err.to_string());
if attempts >= IAM_RETRY_ESCALATION_THRESHOLD { if attempts >= IAM_RETRY_ESCALATION_THRESHOLD {
error!( error!(
event = EVENT_IAM_BOOTSTRAP_RETRY_FAILED, event = EVENT_IAM_BOOTSTRAP_RETRY_FAILED,
@@ -177,6 +178,7 @@ async fn run_iam_recovery_loop<InitFn, FinalizeFn>(
next_retry_secs = next_interval.as_secs(), next_retry_secs = next_interval.as_secs(),
degraded_duration_secs = degraded_since.elapsed().as_secs(), degraded_duration_secs = degraded_since.elapsed().as_secs(),
error = %err, error = %err,
hint,
"IAM bootstrap retry failed; service remains degraded" "IAM bootstrap retry failed; service remains degraded"
); );
} else { } else {
@@ -187,6 +189,7 @@ async fn run_iam_recovery_loop<InitFn, FinalizeFn>(
attempts, attempts,
next_retry_secs = next_interval.as_secs(), next_retry_secs = next_interval.as_secs(),
error = %err, error = %err,
hint,
"IAM bootstrap retry failed; service remains degraded" "IAM bootstrap retry failed; service remains degraded"
); );
} }
@@ -235,6 +238,23 @@ async fn run_iam_recovery_loop<InitFn, FinalizeFn>(
} }
} }
/// Classifies an IAM bootstrap failure into an actionable operator hint, so
/// the degraded-retry logs explain *what to do* instead of only echoing the
/// storage error (rustfs#4304: sequential cold starts left operators
/// guessing why the node stayed degraded).
fn iam_bootstrap_failure_hint(err_text: &str) -> &'static str {
let lower = err_text.to_lowercase();
if lower.contains("quorum") && lower.contains("lock") {
"distributed lock quorum unavailable; waiting for peer nodes' lock RPC endpoints to come online"
} else if lower.contains("read quorum") || lower.contains("erasure") || lower.contains("quorum") {
"storage read quorum not met yet; waiting for enough cluster nodes/disks to come online"
} else if lower.contains("not ready") || lower.contains("not found") {
"storage metadata not initialized yet; retrying automatically"
} else {
"retrying automatically; check storage and peer connectivity if this persists"
}
}
fn initial_retry_interval() -> Duration { fn initial_retry_interval() -> Duration {
// Only honor the test override in debug builds to prevent accidental // Only honor the test override in debug builds to prevent accidental
// production use via environment configuration. // production use via environment configuration.
@@ -376,12 +396,40 @@ pub(crate) async fn init_iam_runtime(
bootstrap_or_defer_iam_init_with_startup_kms(store, readiness, Some(state_manager), Some(ctx)).await bootstrap_or_defer_iam_init_with_startup_kms(store, readiness, Some(state_manager), Some(ctx)).await
} }
#[cfg(test)]
mod hint_tests {
use super::iam_bootstrap_failure_hint;
/// rustfs#4304: the degraded-retry log must tell the operator what the
/// node is waiting on, for each of the observed failure shapes.
#[test]
fn classifies_observed_bootstrap_failures() {
assert!(
iam_bootstrap_failure_hint(
"load group failed: io error: Failed to acquire read lock: Quorum not reached: required 2, achieved 0"
)
.contains("lock quorum"),
"lock-quorum failures must point at peer lock RPC endpoints"
);
assert!(
iam_bootstrap_failure_hint("load group failed: erasure read quorum").contains("storage read quorum"),
"storage-quorum failures must point at missing nodes/disks"
);
assert!(
iam_bootstrap_failure_hint("Storage metadata not ready: probe object not found").contains("not initialized"),
"missing-metadata failures must say the store is still initializing"
);
assert!(
iam_bootstrap_failure_hint("some opaque failure").contains("retrying automatically"),
"unknown failures must still promise the automatic retry"
);
}
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::{ use super::*;
IAM_RETRY_ESCALATION_THRESHOLD, IAM_RETRY_INITIAL_INTERVAL, IAM_RETRY_MAX_INTERVAL, IamBootstrapDisposition, use super::{IAM_RETRY_ESCALATION_THRESHOLD, IAM_RETRY_INITIAL_INTERVAL, IAM_RETRY_MAX_INTERVAL, compute_backoff_interval};
compute_backoff_interval, publish_ready_for_iam_bootstrap_with, run_iam_recovery_loop,
};
use rustfs_common::{GlobalReadiness, SystemStage}; use rustfs_common::{GlobalReadiness, SystemStage};
use std::io::Error; use std::io::Error;
use std::sync::{ use std::sync::{
scripts/check_architecture_migration_rules.sh
+5
View File
@@ -1845,7 +1845,12 @@ if [[ -s "$EXTERNAL_RUNTIME_ECSTORE_COMPAT_BYPASS_HITS_FILE" ]]; then
report_failure "external runtime crates must source ECStore API symbols through their local storage_api boundary: $(paste -sd '; ' "$EXTERNAL_RUNTIME_ECSTORE_COMPAT_BYPASS_HITS_FILE")" report_failure "external runtime crates must source ECStore API symbols through their local storage_api boundary: $(paste -sd '; ' "$EXTERNAL_RUNTIME_ECSTORE_COMPAT_BYPASS_HITS_FILE")"
fi fi
# crates/iam/tests/ecstore_test_compat/mod.rs is a reviewed test-only
# boundary: the sequential-restart regression test (rustfs#4304) needs
# api::global::update_erasure_type to flip into distributed-erasure mode so
# namespace-lock acquisition fails while storage reads stay healthy.
cat >"$GLOBAL_FACADE_BOUNDARY_EXPECTED_FILE" <<'EOF' cat >"$GLOBAL_FACADE_BOUNDARY_EXPECTED_FILE" <<'EOF'
crates/iam/tests/ecstore_test_compat/mod.rs
rustfs/src/storage/storage_api.rs rustfs/src/storage/storage_api.rs
EOF EOF