Files
rustfs/crates/iam/src/lib.rs
T
houseme 717cdd2abd fix(migration): decrypt MinIO IAM & server config on drop-in migration (#4358)
* fix(migration): decrypt MinIO IAM & server config on drop-in migration

MinIO encrypts IAM identity/service-account files and the server config at
rest with a key derived from the root credentials. The drop-in migration
paths read those blobs from the legacy `.minio.sys` bucket and parsed them
as plaintext JSON, so any encrypted blob failed to parse and was silently
skipped with "incompatible format". This is why users migrating from MinIO
kept their buckets/objects/policies but lost users and access keys (#2212).

The IAM load path already knows how to decrypt these blobs (RustFS master
keys plus MinIO-compatible legacy keys derived from the root credentials),
but that logic lived behind a private method and was never used by the
migration paths. Expose it as `rustfs_iam::try_decrypt_iam_blob` and inject
it into both migration paths via a `LegacyBlobDecryptFn` callback (ecstore
cannot depend on the IAM crate, so the closure is wired in the binary crate).
When a blob cannot be decrypted the raw bytes are used as-is, preserving the
previous plaintext-only behavior with no regression.

Also improve object-layer migration observability without changing control
flow: `try_migrate_format` now distinguishes "no legacy format" (a normal
fresh install) from "legacy format present but incompatible", and the caller
logs a loud error before initializing a fresh format that would leave the
existing MinIO objects unreadable. Topology/version skip reasons are promoted
from debug to warn.

Fixes a pre-existing test isolation race by marking
`test_recovery_falls_back_to_default_config_when_blob_stays_corrupt` serial,
since it reads a process-wide env var toggled by a sibling test.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(migration): box FormatV3 in LegacyFormatOutcome to satisfy clippy

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): stabilize concurrent multipart resend lock timeout

concurrent_resend_same_part_commits_one_generation spawns 6 same-part
resends whose cross-disk commits serialize on the per-uploadId commit
lock. Under the full nextest suite the parallel disk load pushes those
serialized commits past the small default lock-acquire timeout (5s),
producing a spurious `Lock(Timeout ...)` unrelated to the property under
test (observed on CI at 5.775s vs ~0.5s in isolation).

Raise RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT to the production default (30s)
for the concurrent-commit section via temp_env, so the regression guard
reflects correctness (exactly one intact generation) rather than disk
latency under CI load. The meaningful assertions are unchanged, and
#[serial] keeps the process-wide env override isolated.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(lock): bound fast-lock notification wait to prevent lost-wakeup stall

The real cause of the concurrent_resend_same_part_commits_one_generation
failures was a lost wakeup in the fast-lock slow path, not disk latency:
raising the acquire timeout to 30s only delayed the failure (it then timed
out at 30s), proving a genuine stall rather than overload.

In acquire_lock_slow_path a waiter that reaches the notification phase did a
single `timeout(remaining, wait_for_write())` spanning the whole acquire
budget, and treated that wait's elapse as a hard `Timeout`. But the release
path only notifies when `writer_waiters > 0`, so if the holder releases in
the gap after the waiter's `try_acquire` fails and before it registers as a
waiter, no notification (and no stored permit, since the pooled `Notify` is
gated) is produced. The waiter then blocks until the deadline even though the
lock is free and stays free — a spurious lock-acquire timeout. The shared
process-wide notify pool makes it worse: a wakeup can be consumed by a waiter
of a different lock hashing to the same slot.

Bound each notification wait (NOTIFY_WAIT_CAP = 50ms) and, on elapse, loop
back and re-`try_acquire` instead of returning `Timeout`; the deadline check
at the top of the loop is the single source of truth for timing out. A
lost/stolen wakeup now degrades to bounded re-polling (acquire within ~50ms
of the lock becoming free) instead of stalling for the whole timeout.
Correctness (mutual exclusion) is unchanged — acquisition still only happens
via `try_acquire_*`.

Add a regression test that reproduces the stall (holder + late waiter across
many keys): it times out without the fix and passes in ~1s with it. Revert
the earlier acquire-timeout workaround in the multipart test now that the
underlying stall is fixed, so it runs under the default timeout again.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-07 16:36:05 +08:00

310 lines
9.6 KiB
Rust

// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::error::{Error, Result};
use manager::IamCache;
use oidc::OidcSys;
use std::sync::{Arc, OnceLock};
use store::object::ObjectStore;
use sys::IamSys;
use tracing::{debug, error, info, instrument, warn};
const LOG_COMPONENT_IAM: &str = "iam";
const LOG_SUBSYSTEM_RUNTIME: &str = "runtime";
const LOG_SUBSYSTEM_OIDC: &str = "oidc";
const EVENT_IAM_STATE: &str = "iam_state";
const EVENT_OIDC_STATE: &str = "oidc_state";
pub mod cache;
pub mod error;
pub mod keyring;
pub mod manager;
pub mod oidc;
pub mod oidc_state;
mod root_credentials;
mod runtime_sources;
mod server_config;
mod storage_api;
pub mod store;
pub mod sys;
pub mod utils;
pub(crate) use storage_api::crate_boundary::{
IAM_CONFIG_ROOT_PREFIX, IamEcstoreError, IamStorageError, IamStore, classify_iam_system_path_failure_reason,
delete_iam_config, is_iam_first_cluster_node_local, read_iam_config_no_lock, read_iam_config_with_metadata, save_iam_config,
save_iam_config_with_opts,
};
pub fn is_root_access_key(access_key: &str) -> bool {
root_credentials::is_root_access_key(access_key)
}
/// Decrypts an at-rest IAM config blob using the same key sources as the IAM load
/// path (RustFS master keys and MinIO-compatible legacy keys derived from the root
/// credentials). Used by the MinIO -> RustFS migration path. See
/// [`store::object::try_decrypt_iam_blob`].
pub use store::object::try_decrypt_iam_blob;
pub(crate) struct IamNotificationPeerErr {
pub(crate) err: Option<IamEcstoreError>,
}
impl From<storage_api::crate_boundary::IamEcstoreNotificationPeerErr> for IamNotificationPeerErr {
fn from(value: storage_api::crate_boundary::IamEcstoreNotificationPeerErr) -> Self {
Self { err: value.err }
}
}
pub(crate) async fn notify_iam_delete_policy(policy_name: &str) -> Vec<IamNotificationPeerErr> {
match runtime_sources::notification_sys() {
Some(notification_sys) => notification_sys
.delete_policy(policy_name)
.await
.into_iter()
.map(Into::into)
.collect(),
None => Vec::new(),
}
}
pub(crate) async fn notify_iam_load_policy(policy_name: &str) -> Vec<IamNotificationPeerErr> {
match runtime_sources::notification_sys() {
Some(notification_sys) => notification_sys
.load_policy(policy_name)
.await
.into_iter()
.map(Into::into)
.collect(),
None => Vec::new(),
}
}
pub(crate) async fn notify_iam_delete_user(access_key: &str) -> Vec<IamNotificationPeerErr> {
match runtime_sources::notification_sys() {
Some(notification_sys) => notification_sys
.delete_user(access_key)
.await
.into_iter()
.map(Into::into)
.collect(),
None => Vec::new(),
}
}
pub(crate) async fn notify_iam_load_user(access_key: &str, temp: bool) -> Vec<IamNotificationPeerErr> {
match runtime_sources::notification_sys() {
Some(notification_sys) => notification_sys
.load_user(access_key, temp)
.await
.into_iter()
.map(Into::into)
.collect(),
None => Vec::new(),
}
}
pub(crate) async fn notify_iam_load_service_account(access_key: &str) -> Vec<IamNotificationPeerErr> {
match runtime_sources::notification_sys() {
Some(notification_sys) => notification_sys
.load_service_account(access_key)
.await
.into_iter()
.map(Into::into)
.collect(),
None => Vec::new(),
}
}
pub(crate) async fn notify_iam_delete_service_account(access_key: &str) -> Vec<IamNotificationPeerErr> {
match runtime_sources::notification_sys() {
Some(notification_sys) => notification_sys
.delete_service_account(access_key)
.await
.into_iter()
.map(Into::into)
.collect(),
None => Vec::new(),
}
}
pub(crate) async fn notify_iam_load_group(group: &str) -> Vec<IamNotificationPeerErr> {
match runtime_sources::notification_sys() {
Some(notification_sys) => notification_sys.load_group(group).await.into_iter().map(Into::into).collect(),
None => Vec::new(),
}
}
pub(crate) async fn notify_iam_load_policy_mapping(
user_or_group: &str,
user_type: u64,
is_group: bool,
) -> Vec<IamNotificationPeerErr> {
match runtime_sources::notification_sys() {
Some(notification_sys) => notification_sys
.load_policy_mapping(user_or_group, user_type, is_group)
.await
.into_iter()
.map(Into::into)
.collect(),
None => Vec::new(),
}
}
static IAM_SYS: OnceLock<Arc<IamSys<ObjectStore>>> = OnceLock::new();
static OIDC_SYS: OnceLock<Arc<OidcSys>> = OnceLock::new();
#[instrument(skip(ecstore))]
pub async fn init_iam_sys(ecstore: Arc<IamStore>) -> Result<()> {
if IAM_SYS.get().is_some() {
info!(
event = EVENT_IAM_STATE,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_RUNTIME,
state = "already_initialized",
"IAM runtime already initialized"
);
return Ok(());
}
info!(
event = EVENT_IAM_STATE,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_RUNTIME,
state = "starting",
"IAM runtime starting"
);
// 1. Create the persistent storage adapter
let storage_adapter = ObjectStore::new(ecstore);
// 2. Create the cache manager.
// The `new` method now performs a blocking initial load from disk.
let cache_manager = IamCache::new(storage_adapter).await?;
// 3. Construct the system interface
let iam_instance = Arc::new(IamSys::new(cache_manager));
// 4. Securely set the global singleton
if IAM_SYS.set(iam_instance).is_err() {
error!(
event = EVENT_IAM_STATE,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_RUNTIME,
state = "singleton_set_failed",
"IAM runtime singleton set failed"
);
return Err(Error::IamSysAlreadyInitialized);
}
info!(
event = EVENT_IAM_STATE,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_RUNTIME,
state = "ready",
"IAM runtime ready"
);
Ok(())
}
#[inline]
pub fn get() -> Result<Arc<IamSys<ObjectStore>>> {
let sys = IAM_SYS.get().map(Arc::clone).ok_or(Error::IamSysNotInitialized)?;
// Double-check the internal readiness state. The OnceLock is only set
// after initialization and data loading complete, so this is a defensive
// guard to ensure callers never operate on a partially initialized system.
if !sys.is_ready() {
return Err(Error::IamSysNotInitialized);
}
Ok(sys)
}
pub fn get_global_iam_sys() -> Option<Arc<IamSys<ObjectStore>>> {
IAM_SYS.get().cloned()
}
/// Initialize the global OIDC system. Non-fatal if no OIDC providers are configured.
pub async fn init_oidc_sys() -> Result<()> {
if OIDC_SYS.get().is_some() {
debug!(
event = EVENT_OIDC_STATE,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
state = "already_initialized",
"OIDC runtime already initialized"
);
return Ok(());
}
debug!(
event = EVENT_OIDC_STATE,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
state = "starting",
"OIDC runtime starting"
);
let oidc_sys = match OidcSys::new().await {
Ok(sys) => {
if sys.has_providers() {
debug!(
event = EVENT_OIDC_STATE,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
provider_count = sys.list_providers().len(),
state = "ready",
"OIDC runtime ready"
);
} else {
debug!(
event = EVENT_OIDC_STATE,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
state = "empty",
"OIDC runtime has no providers"
);
}
sys
}
Err(e) => {
warn!(
event = EVENT_OIDC_STATE,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
state = "init_failed_non_fatal",
error = %e,
"OIDC runtime initialization failed"
);
OidcSys::empty().map_err(Error::StringError)?
}
};
if OIDC_SYS.set(Arc::new(oidc_sys)).is_err() {
warn!(
event = EVENT_OIDC_STATE,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
state = "singleton_set_race",
"OIDC runtime singleton set raced"
);
}
Ok(())
}
/// Get the global OIDC system.
pub fn get_oidc() -> Option<Arc<OidcSys>> {
OIDC_SYS.get().cloned()
}