mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-28 00:58:59 +00:00
7001373316
* fix(iam): load IAM bootstrap snapshot without namespace locks
IAM bootstrap (init_iam_sys -> load_all) read every config object with
default ObjectOptions (no_lock=false), so each read acquired a
distributed namespace read lock. Lock quorum is counted over cluster
nodes and unreachable peers are hard failures, so during a sequential
restart the very first read failed with
"Quorum not reached: required 2, achieved 0" and IAM could not come up
until enough peers' lock RPC surfaces converged - even when the storage
read quorum was already satisfiable (rustfs#4304).
Extend the startup contract from rustfs#4056 ("startup metadata I/O must
not require namespace locks") to the IAM bootstrap path:
- Introduce LoadMode {Locked, BootstrapNoLock} and plumb it through the
load_all chain (groups, users, policies, mapped policies and their
concurrent variants) down to the storage read options.
- load_all now performs all reads with no_lock=true; on-line
single-object loads via the Store trait keep locked semantics.
- Fail-closed behavior is unchanged: any loader error still aborts the
whole snapshot load.
Safety: config objects are atomic whole-object writes, so a lock-free
read only observes an old or a new value; staleness is bounded by the
existing periodic IAM reload. Listing (walk) never took namespace locks,
and maybe_schedule_lazy_rewrite stays a best-effort background task.
Verification:
- cargo test -p rustfs-iam --lib (153 passed, incl. new LoadMode tests)
- cargo clippy -p rustfs-iam --all-targets
- cargo check -p rustfs
- make pre-commit
Ref: rustfs#4304; tracking rustfs/backlog#884, rustfs/backlog#885
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(iam): sequential-restart regression test for lock-free bootstrap
Add an integration test reproducing the rustfs#4304 failure mode against
a real 4-disk temp-dir ECStore:
- Seed IAM group data in single-node mode, then flip the runtime into
distributed-erasure mode. new_ns_lock now builds a distributed lock
over the set's (empty) lock-client list, so every namespace-locked
read fails exactly like a sequential restart with unreachable peers
(lock quorum unavailable, storage read quorum healthy).
- Assert the locked load_group path fails in that state, while the
bulk snapshot load_all (no_lock plumbing from the previous commit)
succeeds, and the data survives intact once single-node mode is
restored.
Reverse-verified: temporarily switching load_all back to the locked
mode makes the test fail, so it genuinely guards the contract.
Verification:
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- cargo test -p rustfs-iam --lib
Ref: rustfs#4304; tracking rustfs/backlog#886
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(iam): route test ECStore imports through ecstore_test_compat boundary
The new integration test imported rustfs_ecstore facade paths directly,
tripping three architecture migration rules. Move every ECStore import
behind crates/iam/tests/ecstore_test_compat/mod.rs (the sanctioned
test-compat pattern), and register that module as a reviewed test-only
global-facade boundary in check_architecture_migration_rules.sh: the
sequential-restart regression test needs api::global::update_erasure_type
to flip into distributed-erasure mode for lock-quorum fault injection.
Verification:
- ./scripts/check_architecture_migration_rules.sh
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- make pre-commit
Co-Authored-By: heihutu <heihutu@gmail.com>
* feat(server): expose readiness blocking reason + rolling-restart runbook
Operators hitting the rustfs#4304 sequential cold start could not tell
from the outside why a node stayed unavailable. Three additions:
- The readiness gate's 503 now names the blocking dependency in both the
body ("Service not ready: waiting for storage_quorum") and a new
x-rustfs-readiness-pending header (storage_quorum | iam |
startup_finalization), derived from the current startup stage.
/health/ready already returned details + degradedReasons; this covers
the plain S3 requests that hit the gate.
- IAM bootstrap retry logs now carry an actionable `hint` field that
classifies the failure (storage read quorum vs lock quorum vs
uninitialized metadata) instead of only echoing the storage error.
- New docs/operations/rolling-restart.md runbook: correct rolling
restart procedure, sequential cold-start expectations (degraded ->
auto-recovery), readiness signal reference, and
RUSTFS_STARTUP_READINESS_MAX_WAIT_SECS guidance.
Verification:
- cargo test -p rustfs --lib -- hint_tests service_not_ready readiness_pending
- make pre-commit
Ref: rustfs#4304; tracking rustfs/backlog#887
Co-Authored-By: heihutu <heihutu@gmail.com>
* upgrade deps version and improve import
* feat(iam): notification-path cache refreshes read without namespace locks (#4368)
P3 step 1 of rustfs/backlog#884 (scoped down from full MinIO readConfig
alignment after review): cross-node notification handlers
(group/policy/policy-mapping/user) refresh the local IAM cache with
single-object reads that previously took distributed namespace read
locks. These refreshes are asynchronous, best-effort, and already
stale-tolerant (the periodic reload converges them), so a node-counted
lock quorum failure or lock RPC hiccup on a peer must not fail them —
the same rationale as the lock-free bootstrap load_all (rustfs#4304).
- Store trait: add load_user_no_lock / load_group_no_lock /
load_policy_doc_no_lock / load_mapped_policy_no_lock with defaults
forwarding to the locked variants, so existing implementations and
test mocks keep their behavior.
- ObjectStore overrides them via the existing LoadMode::BootstrapNoLock
plumbing. Deletions triggered by the handlers keep locked writes.
- manager.rs: the four *_notification_handler paths (8 call sites)
switch to the lock-free variants.
- Integration test: while the lock quorum is unavailable (DistErasure
with empty lockers), load_group_no_lock must succeed exactly where
the locked load_group fails.
Request-path loads (check_key, verify_temp_user_persistence) and admin
write-then-reload paths intentionally stay locked: load_user_identity
embeds expiry deletions, so those need the side-effect extraction
tracked in rustfs/backlog#884 before going lock-free.
Verification:
- cargo test -p rustfs-iam --lib (156 passed)
- cargo test -p rustfs-iam --test iam_bootstrap_no_lock_test
- make pre-commit
Ref: rustfs/backlog#884, rustfs#4304
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(server): drop unused iam_bootstrap_failure_hint import in tests
The hint tests live in their own hint_tests module with a local import;
the stale re-import in mod tests failed clippy's -D warnings on the
Test and Lint CI variants.
Verification:
- cargo clippy -p rustfs --all-targets
Co-Authored-By: heihutu <heihutu@gmail.com>
* fix
---------
Co-authored-by: heihutu <heihutu@gmail.com>
257 lines
9.0 KiB
Rust
257 lines
9.0 KiB
Rust
// Copyright 2024 RustFS Team
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
pub mod object;
|
|
|
|
use crate::cache::Cache;
|
|
use crate::error::Result;
|
|
use rustfs_policy::{auth::UserIdentity, policy::PolicyDoc};
|
|
use serde::{Deserialize, Serialize, de::DeserializeOwned};
|
|
use std::collections::{HashMap, HashSet};
|
|
use time::OffsetDateTime;
|
|
|
|
#[async_trait::async_trait]
|
|
pub trait Store: Clone + Send + Sync + 'static {
|
|
fn has_watcher(&self) -> bool;
|
|
async fn save_iam_config<Item: Serialize + Send>(&self, item: Item, path: impl AsRef<str> + Send) -> Result<()>;
|
|
async fn load_iam_config<Item: DeserializeOwned>(&self, path: impl AsRef<str> + Send) -> Result<Item>;
|
|
async fn delete_iam_config(&self, path: impl AsRef<str> + Send) -> Result<()>;
|
|
|
|
async fn save_user_identity(&self, name: &str, user_type: UserType, item: UserIdentity, ttl: Option<usize>) -> Result<()>;
|
|
async fn delete_user_identity(&self, name: &str, user_type: UserType) -> Result<()>;
|
|
async fn load_user_identity(&self, name: &str, user_type: UserType) -> Result<UserIdentity>;
|
|
|
|
async fn load_user(&self, name: &str, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()>;
|
|
async fn load_users(&self, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()>;
|
|
async fn load_secret_key(&self, name: &str, user_type: UserType) -> Result<String>;
|
|
|
|
async fn save_group_info(&self, name: &str, item: GroupInfo) -> Result<()>;
|
|
async fn delete_group_info(&self, name: &str) -> Result<()>;
|
|
async fn load_group(&self, name: &str, m: &mut HashMap<String, GroupInfo>) -> Result<()>;
|
|
async fn load_groups(&self, m: &mut HashMap<String, GroupInfo>) -> Result<()>;
|
|
|
|
async fn save_policy_doc(&self, name: &str, item: PolicyDoc) -> Result<()>;
|
|
async fn delete_policy_doc(&self, name: &str) -> Result<()>;
|
|
async fn load_policy(&self, name: &str) -> Result<PolicyDoc>;
|
|
async fn load_policy_doc(&self, name: &str, m: &mut HashMap<String, PolicyDoc>) -> Result<()>;
|
|
async fn load_policy_docs(&self, m: &mut HashMap<String, PolicyDoc>) -> Result<()>;
|
|
|
|
async fn save_mapped_policy(
|
|
&self,
|
|
name: &str,
|
|
user_type: UserType,
|
|
is_group: bool,
|
|
item: MappedPolicy,
|
|
ttl: Option<usize>,
|
|
) -> Result<()>;
|
|
async fn delete_mapped_policy(&self, name: &str, user_type: UserType, is_group: bool) -> Result<()>;
|
|
async fn load_mapped_policy(
|
|
&self,
|
|
name: &str,
|
|
user_type: UserType,
|
|
is_group: bool,
|
|
m: &mut HashMap<String, MappedPolicy>,
|
|
) -> Result<()>;
|
|
async fn load_mapped_policies(
|
|
&self,
|
|
user_type: UserType,
|
|
is_group: bool,
|
|
m: &mut HashMap<String, MappedPolicy>,
|
|
) -> Result<()>;
|
|
|
|
async fn load_all(&self, cache: &Cache) -> Result<()>;
|
|
|
|
// Lock-free variants used by the cross-node notification handlers.
|
|
//
|
|
// Notification-path cache refreshes are asynchronous, best-effort, and
|
|
// already tolerate stale values (the periodic reload converges them), so
|
|
// they must not depend on the node-counted namespace-lock quorum — the
|
|
// same rationale as the lock-free bootstrap `load_all` (rustfs#4304;
|
|
// MinIO's readConfig takes no distributed lock either). The defaults
|
|
// forward to the locked variants so existing `Store` implementations
|
|
// (including test mocks) keep their behavior; `ObjectStore` overrides
|
|
// them with lock-free reads.
|
|
async fn load_user_no_lock(&self, name: &str, user_type: UserType, m: &mut HashMap<String, UserIdentity>) -> Result<()> {
|
|
self.load_user(name, user_type, m).await
|
|
}
|
|
async fn load_group_no_lock(&self, name: &str, m: &mut HashMap<String, GroupInfo>) -> Result<()> {
|
|
self.load_group(name, m).await
|
|
}
|
|
async fn load_policy_doc_no_lock(&self, name: &str, m: &mut HashMap<String, PolicyDoc>) -> Result<()> {
|
|
self.load_policy_doc(name, m).await
|
|
}
|
|
async fn load_mapped_policy_no_lock(
|
|
&self,
|
|
name: &str,
|
|
user_type: UserType,
|
|
is_group: bool,
|
|
m: &mut HashMap<String, MappedPolicy>,
|
|
) -> Result<()> {
|
|
self.load_mapped_policy(name, user_type, is_group, m).await
|
|
}
|
|
}
|
|
|
|
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
|
|
pub enum UserType {
|
|
Svc,
|
|
Sts,
|
|
Reg,
|
|
None,
|
|
}
|
|
|
|
impl UserType {
|
|
pub fn prefix(&self) -> &'static str {
|
|
match self {
|
|
UserType::Svc => "service-accounts/",
|
|
UserType::Sts => "sts/",
|
|
UserType::Reg => "users/",
|
|
UserType::None => "",
|
|
}
|
|
}
|
|
pub fn to_u64(&self) -> u64 {
|
|
match self {
|
|
UserType::Svc => 1,
|
|
UserType::Sts => 2,
|
|
UserType::Reg => 3,
|
|
UserType::None => 0,
|
|
}
|
|
}
|
|
|
|
pub fn from_u64(u64: u64) -> Option<Self> {
|
|
match u64 {
|
|
1 => Some(UserType::Svc),
|
|
2 => Some(UserType::Sts),
|
|
3 => Some(UserType::Reg),
|
|
0 => Some(UserType::None),
|
|
_ => None,
|
|
}
|
|
}
|
|
}
|
|
|
|
#[derive(Serialize, Deserialize, Clone)]
|
|
pub struct MappedPolicy {
|
|
pub version: i64,
|
|
/// policy, legacy: policies. Serialize as policy.
|
|
#[serde(rename = "policy", alias = "policies")]
|
|
pub policies: String,
|
|
/// updatedAt (RFC3339), legacy: update_at. Serialize as updatedAt.
|
|
#[serde(rename = "updatedAt", alias = "update_at", with = "rustfs_policy::serde_datetime")]
|
|
pub update_at: OffsetDateTime,
|
|
}
|
|
|
|
impl Default for MappedPolicy {
|
|
fn default() -> Self {
|
|
Self {
|
|
version: 0,
|
|
policies: "".to_owned(),
|
|
update_at: OffsetDateTime::now_utc(),
|
|
}
|
|
}
|
|
}
|
|
|
|
impl MappedPolicy {
|
|
pub fn new(policy: &str) -> Self {
|
|
Self {
|
|
version: 1,
|
|
policies: policy.to_owned(),
|
|
update_at: OffsetDateTime::now_utc(),
|
|
}
|
|
}
|
|
|
|
pub fn to_slice(&self) -> Vec<String> {
|
|
self.policies
|
|
.split(",")
|
|
.filter(|v| !v.trim().is_empty())
|
|
.map(|v| v.to_string())
|
|
.collect()
|
|
}
|
|
|
|
pub fn policy_set(&self) -> HashSet<String> {
|
|
self.policies
|
|
.split(",")
|
|
.filter(|v| !v.trim().is_empty())
|
|
.map(|v| v.to_string())
|
|
.collect()
|
|
}
|
|
}
|
|
|
|
#[derive(Serialize, Deserialize, Clone, Debug, Default)]
|
|
pub struct GroupInfo {
|
|
pub version: i64,
|
|
pub status: String,
|
|
pub members: Vec<String>,
|
|
/// updatedAt (RFC3339), legacy: update_at. Serialize as updatedAt.
|
|
#[serde(
|
|
rename = "updatedAt",
|
|
alias = "update_at",
|
|
default,
|
|
with = "rustfs_policy::serde_datetime::option"
|
|
)]
|
|
pub update_at: Option<OffsetDateTime>,
|
|
}
|
|
|
|
impl GroupInfo {
|
|
pub fn new(members: Vec<String>) -> Self {
|
|
Self {
|
|
version: 1,
|
|
status: "enabled".to_owned(),
|
|
members,
|
|
update_at: Some(OffsetDateTime::now_utc()),
|
|
}
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::{GroupInfo, MappedPolicy};
|
|
|
|
/// uses RFC3339 for updatedAt. MappedPolicy must serialize as RFC3339.
|
|
#[test]
|
|
fn test_mapped_policy_timestamps_serialize_as_rfc3339() {
|
|
let mp = MappedPolicy::new("readwrite");
|
|
let json = serde_json::to_string(&mp).expect("serialize");
|
|
assert!(json.contains('T'), "MappedPolicy updatedAt should be RFC3339; got: {}", json);
|
|
assert!(
|
|
json.contains('Z') || json.contains("+00:00"),
|
|
"MappedPolicy updatedAt should be RFC3339; got: {}",
|
|
json
|
|
);
|
|
}
|
|
|
|
/// Deserialize MappedPolicy from JSON (RFC3339 updatedAt).
|
|
#[test]
|
|
fn test_mapped_policy_deserialize_minio_style_rfc3339() {
|
|
let minio_style = r#"{"version":1,"policy":"readwrite","updatedAt":"2025-03-07T12:00:00Z"}"#;
|
|
let mp: MappedPolicy = serde_json::from_str(minio_style).expect("deserialize");
|
|
assert_eq!(mp.policies, "readwrite");
|
|
}
|
|
|
|
/// GroupInfo updatedAt: uses RFC3339.
|
|
#[test]
|
|
fn test_group_info_timestamps_serialize_as_rfc3339() {
|
|
let g = GroupInfo::new(vec!["u1".to_string()]);
|
|
let json = serde_json::to_string(&g).expect("serialize");
|
|
assert!(json.contains('T'), "GroupInfo updatedAt should be RFC3339; got: {}", json);
|
|
}
|
|
|
|
/// Deserialize GroupInfo from JSON (RFC3339 updatedAt).
|
|
#[test]
|
|
fn test_group_info_deserialize_minio_style_rfc3339() {
|
|
let minio_style = r#"{"version":1,"status":"enabled","members":["u1"],"updatedAt":"2025-03-07T12:00:00Z"}"#;
|
|
let g: GroupInfo = serde_json::from_str(minio_style).expect("deserialize");
|
|
assert_eq!(g.members, ["u1"]);
|
|
assert!(g.update_at.is_some());
|
|
}
|
|
}
|