Files
rustfs/crates/iam/src/store.rs
T
houseme 7001373316 fix(iam): load IAM bootstrap snapshot without namespace locks (#4363)
* fix(iam): load IAM bootstrap snapshot without namespace locks

IAM bootstrap (init_iam_sys -> load_all) read every config object with
default ObjectOptions (no_lock=false), so each read acquired a
distributed namespace read lock. Lock quorum is counted over cluster
nodes and unreachable peers are hard failures, so during a sequential
restart the very first read failed with
"Quorum not reached: required 2, achieved 0" and IAM could not come up
until enough peers' lock RPC surfaces converged - even when the storage
read quorum was already satisfiable (rustfs#4304).

Extend the startup contract from rustfs#4056 ("startup metadata I/O must
not require namespace locks") to the IAM bootstrap path:

- Introduce LoadMode {Locked, BootstrapNoLock} and plumb it through the
  load_all chain (groups, users, policies, mapped policies and their
  concurrent variants) down to the storage read options.
- load_all now performs all reads with no_lock=true; on-line
  single-object loads via the Store trait keep locked semantics.
- Fail-closed behavior is unchanged: any loader error still aborts the
  whole snapshot load.

Safety: config objects are atomic whole-object writes, so a lock-free
read only observes an old or a new value; staleness is bounded by the
existing periodic IAM reload. Listing (walk) never took namespace locks,
and maybe_schedule_lazy_rewrite stays a best-effort background task.

Verification:
- cargo test -p rustfs-iam --lib (153 passed, incl. new LoadMode tests)
- cargo clippy -p rustfs-iam --all-targets
- cargo check -p rustfs
- make pre-commit

Ref: rustfs#4304; tracking rustfs/backlog#884, rustfs/backlog#885

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(iam): sequential-restart regression test for lock-free bootstrap

Add an integration test reproducing the rustfs#4304 failure mode against
a real 4-disk temp-dir ECStore:

- Seed IAM group data in single-node mode, then flip the runtime into
  distributed-erasure mode. new_ns_lock now builds a distributed lock
  over the set's (empty) lock-client list, so every namespace-locked
  read fails exactly like a sequential restart with unreachable peers
  (lock quorum unavailable, storage read quorum healthy).
- Assert the locked load_group path fails in that state, while the
  bulk snapshot load_all (no_lock plumbing from the previous commit)
  succeeds, and the data survives intact once single-node mode is
  restored.

Reverse-verified: temporarily switching load_all back to the locked
mode makes the test fail, so it genuinely guards the contract.

Verification:
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- cargo test -p rustfs-iam --lib

Ref: rustfs#4304; tracking rustfs/backlog#886

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(iam): route test ECStore imports through ecstore_test_compat boundary

The new integration test imported rustfs_ecstore facade paths directly,
tripping three architecture migration rules. Move every ECStore import
behind crates/iam/tests/ecstore_test_compat/mod.rs (the sanctioned
test-compat pattern), and register that module as a reviewed test-only
global-facade boundary in check_architecture_migration_rules.sh: the
sequential-restart regression test needs api::global::update_erasure_type
to flip into distributed-erasure mode for lock-quorum fault injection.

Verification:
- ./scripts/check_architecture_migration_rules.sh
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- make pre-commit

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(server): expose readiness blocking reason + rolling-restart runbook

Operators hitting the rustfs#4304 sequential cold start could not tell
from the outside why a node stayed unavailable. Three additions:

- The readiness gate's 503 now names the blocking dependency in both the
  body ("Service not ready: waiting for storage_quorum") and a new
  x-rustfs-readiness-pending header (storage_quorum | iam |
  startup_finalization), derived from the current startup stage.
  /health/ready already returned details + degradedReasons; this covers
  the plain S3 requests that hit the gate.
- IAM bootstrap retry logs now carry an actionable `hint` field that
  classifies the failure (storage read quorum vs lock quorum vs
  uninitialized metadata) instead of only echoing the storage error.
- New docs/operations/rolling-restart.md runbook: correct rolling
  restart procedure, sequential cold-start expectations (degraded ->
  auto-recovery), readiness signal reference, and
  RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS guidance.

Verification:
- cargo test -p rustfs --lib -- hint_tests service_not_ready readiness_pending
- make pre-commit

Ref: rustfs#4304; tracking rustfs/backlog#887

Co-Authored-By: heihutu <heihutu@gmail.com>

* upgrade deps version and improve import

* feat(iam): notification-path cache refreshes read without namespace locks (#4368)

P3 step 1 of rustfs/backlog#884 (scoped down from full MinIO readConfig
alignment after review): cross-node notification handlers
(group/policy/policy-mapping/user) refresh the local IAM cache with
single-object reads that previously took distributed namespace read
locks. These refreshes are asynchronous, best-effort, and already
stale-tolerant (the periodic reload converges them), so a node-counted
lock quorum failure or lock RPC hiccup on a peer must not fail them —
the same rationale as the lock-free bootstrap load_all (rustfs#4304).

- Store trait: add load_user_no_lock / load_group_no_lock /
  load_policy_doc_no_lock / load_mapped_policy_no_lock with defaults
  forwarding to the locked variants, so existing implementations and
  test mocks keep their behavior.
- ObjectStore overrides them via the existing LoadMode::BootstrapNoLock
  plumbing. Deletions triggered by the handlers keep locked writes.
- manager.rs: the four *_notification_handler paths (8 call sites)
  switch to the lock-free variants.
- Integration test: while the lock quorum is unavailable (DistErasure
  with empty lockers), load_group_no_lock must succeed exactly where
  the locked load_group fails.

Request-path loads (check_key, verify_temp_user_persistence) and admin
write-then-reload paths intentionally stay locked: load_user_identity
embeds expiry deletions, so those need the side-effect extraction
tracked in rustfs/backlog#884 before going lock-free.

Verification:
- cargo test -p rustfs-iam --lib (156 passed)
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- make pre-commit

Ref: rustfs/backlog#884, rustfs#4304

Co-authored-by: heihutu <heihutu@gmail.com>

* fix(server): drop unused iam_bootstrap_failure_hint import in tests

The hint tests live in their own hint_tests module with a local import;
the stale re-import in mod tests failed clippy's -D warnings on the
Test and Lint CI variants.

Verification:
- cargo clippy -p rustfs --all-targets

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-07 20:07:09 +08:00

257 lines
9.0 KiB
Rust

// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
pub mod object;
use crate::cache::Cache;
use crate::error::Result;
use rustfs_policy::{auth::UserIdentity, policy::PolicyDoc};
use serde::{Deserialize, Serialize, de::DeserializeOwned};
use std::collections::{HashMap, HashSet};
use time::OffsetDateTime;
#[async_trait::async_trait]
pub trait Store: Clone + Send + Sync + 'static {
fn has_watcher(&self) -> bool;
async fn save_iam_config<Item: Serialize + Send>(&self, item: Item, path: impl AsRef<str> + Send) -> Result<()>;
async fn load_iam_config<Item: DeserializeOwned>(&self, path: impl AsRef<str> + Send) -> Result<Item>;
async fn delete_iam_config(&self, path: impl AsRef<str> + Send) -> Result<()>;
async fn save_user_identity(&self, name: &str, user_type: UserType, item: UserIdentity, ttl: Option<usize>) -> Result<()>;
async fn delete_user_identity(&self, name: &str, user_type: UserType) -> Result<()>;
async fn load_user_identity(&self, name: &str, user_type: UserType) -> Result<UserIdentity>;
async fn load_user(&self, name: &str, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()>;
async fn load_users(&self, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()>;
async fn load_secret_key(&self, name: &str, user_type: UserType) -> Result<String>;
async fn save_group_info(&self, name: &str, item: GroupInfo) -> Result<()>;
async fn delete_group_info(&self, name: &str) -> Result<()>;
async fn load_group(&self, name: &str, m: &mut HashMap<String, GroupInfo>) -> Result<()>;
async fn load_groups(&self, m: &mut HashMap<String, GroupInfo>) -> Result<()>;
async fn save_policy_doc(&self, name: &str, item: PolicyDoc) -> Result<()>;
async fn delete_policy_doc(&self, name: &str) -> Result<()>;
async fn load_policy(&self, name: &str) -> Result<PolicyDoc>;
async fn load_policy_doc(&self, name: &str, m: &mut HashMap<String, PolicyDoc>) -> Result<()>;
async fn load_policy_docs(&self, m: &mut HashMap<String, PolicyDoc>) -> Result<()>;
async fn save_mapped_policy(
&self,
name: &str,
user_type: UserType,
is_group: bool,
item: MappedPolicy,
ttl: Option<usize>,
) -> Result<()>;
async fn delete_mapped_policy(&self, name: &str, user_type: UserType, is_group: bool) -> Result<()>;
async fn load_mapped_policy(
&self,
name: &str,
user_type: UserType,
is_group: bool,
m: &mut HashMap<String, MappedPolicy>,
) -> Result<()>;
async fn load_mapped_policies(
&self,
user_type: UserType,
is_group: bool,
m: &mut HashMap<String, MappedPolicy>,
) -> Result<()>;
async fn load_all(&self, cache: &Cache) -> Result<()>;
// Lock-free variants used by the cross-node notification handlers.
//
// Notification-path cache refreshes are asynchronous, best-effort, and
// already tolerate stale values (the periodic reload converges them), so
// they must not depend on the node-counted namespace-lock quorum — the
// same rationale as the lock-free bootstrap `load_all` (rustfs#4304;
// MinIO's readConfig takes no distributed lock either). The defaults
// forward to the locked variants so existing `Store` implementations
// (including test mocks) keep their behavior; `ObjectStore` overrides
// them with lock-free reads.
async fn load_user_no_lock(&self, name: &str, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()> {
self.load_user(name, user_type, m).await
}
async fn load_group_no_lock(&self, name: &str, m: &mut HashMap<String, GroupInfo>) -> Result<()> {
self.load_group(name, m).await
}
async fn load_policy_doc_no_lock(&self, name: &str, m: &mut HashMap<String, PolicyDoc>) -> Result<()> {
self.load_policy_doc(name, m).await
}
async fn load_mapped_policy_no_lock(
&self,
name: &str,
user_type: UserType,
is_group: bool,
m: &mut HashMap<String, MappedPolicy>,
) -> Result<()> {
self.load_mapped_policy(name, user_type, is_group, m).await
}
}
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
pub enum UserType {
Svc,
Sts,
Reg,
None,
}
impl UserType {
pub fn prefix(&self) -> &'static str {
match self {
UserType::Svc => "service-accounts/",
UserType::Sts => "sts/",
UserType::Reg => "users/",
UserType::None => "",
}
}
pub fn to_u64(&self) -> u64 {
match self {
UserType::Svc => 1,
UserType::Sts => 2,
UserType::Reg => 3,
UserType::None => 0,
}
}
pub fn from_u64(u64: u64) -> Option<Self> {
match u64 {
1 => Some(UserType::Svc),
2 => Some(UserType::Sts),
3 => Some(UserType::Reg),
0 => Some(UserType::None),
_ => None,
}
}
}
#[derive(Serialize, Deserialize, Clone)]
pub struct MappedPolicy {
pub version: i64,
/// policy, legacy: policies. Serialize as policy.
#[serde(rename = "policy", alias = "policies")]
pub policies: String,
/// updatedAt (RFC3339), legacy: update_at. Serialize as updatedAt.
#[serde(rename = "updatedAt", alias = "update_at", with = "rustfs_policy::serde_datetime")]
pub update_at: OffsetDateTime,
}
impl Default for MappedPolicy {
fn default() -> Self {
Self {
version: 0,
policies: "".to_owned(),
update_at: OffsetDateTime::now_utc(),
}
}
}
impl MappedPolicy {
pub fn new(policy: &str) -> Self {
Self {
version: 1,
policies: policy.to_owned(),
update_at: OffsetDateTime::now_utc(),
}
}
pub fn to_slice(&self) -> Vec<String> {
self.policies
.split(",")
.filter(|v| !v.trim().is_empty())
.map(|v| v.to_string())
.collect()
}
pub fn policy_set(&self) -> HashSet<String> {
self.policies
.split(",")
.filter(|v| !v.trim().is_empty())
.map(|v| v.to_string())
.collect()
}
}
#[derive(Serialize, Deserialize, Clone, Debug, Default)]
pub struct GroupInfo {
pub version: i64,
pub status: String,
pub members: Vec<String>,
/// updatedAt (RFC3339), legacy: update_at. Serialize as updatedAt.
#[serde(
rename = "updatedAt",
alias = "update_at",
default,
with = "rustfs_policy::serde_datetime::option"
)]
pub update_at: Option<OffsetDateTime>,
}
impl GroupInfo {
pub fn new(members: Vec<String>) -> Self {
Self {
version: 1,
status: "enabled".to_owned(),
members,
update_at: Some(OffsetDateTime::now_utc()),
}
}
}
#[cfg(test)]
mod tests {
use super::{GroupInfo, MappedPolicy};
/// uses RFC3339 for updatedAt. MappedPolicy must serialize as RFC3339.
#[test]
fn test_mapped_policy_timestamps_serialize_as_rfc3339() {
let mp = MappedPolicy::new("readwrite");
let json = serde_json::to_string(&mp).expect("serialize");
assert!(json.contains('T'), "MappedPolicy updatedAt should be RFC3339; got: {}", json);
assert!(
json.contains('Z') || json.contains("+00:00"),
"MappedPolicy updatedAt should be RFC3339; got: {}",
json
);
}
/// Deserialize MappedPolicy from JSON (RFC3339 updatedAt).
#[test]
fn test_mapped_policy_deserialize_minio_style_rfc3339() {
let minio_style = r#"{"version":1,"policy":"readwrite","updatedAt":"2025-03-07T12:00:00Z"}"#;
let mp: MappedPolicy = serde_json::from_str(minio_style).expect("deserialize");
assert_eq!(mp.policies, "readwrite");
}
/// GroupInfo updatedAt: uses RFC3339.
#[test]
fn test_group_info_timestamps_serialize_as_rfc3339() {
let g = GroupInfo::new(vec!["u1".to_string()]);
let json = serde_json::to_string(&g).expect("serialize");
assert!(json.contains('T'), "GroupInfo updatedAt should be RFC3339; got: {}", json);
}
/// Deserialize GroupInfo from JSON (RFC3339 updatedAt).
#[test]
fn test_group_info_deserialize_minio_style_rfc3339() {
let minio_style = r#"{"version":1,"status":"enabled","members":["u1"],"updatedAt":"2025-03-07T12:00:00Z"}"#;
let g: GroupInfo = serde_json::from_str(minio_style).expect("deserialize");
assert_eq!(g.members, ["u1"]);
assert!(g.update_at.is_some());
}
}