fix(iam): prevent transient IAM walk timeout from crashing startup (#3188)

* fix(iam): prevent transient IAM walk timeout from crashing startup

  IAM startup performs a blocking full metadata walk on `.rustfs.sys/config/iam/`.
  When that distributed walk times out (e.g. disk pressure after cluster reboot),
  the old code treated the failure as fatal and exited the process, causing a
  systemd restart loop.

  Changes:
  - Add `startup_iam.rs`: attempt IAM init, enter degraded mode on failure,
    spawn background retry task with exponential backoff (5s→10s→20s→30s cap)
  - Log level escalates to ERROR after 12 retries (~5 min) to aid diagnosis
  - `/health/ready` returns 503 until IAM recovers; IAM-dependent ops return
    `IamSysNotInitialized` (existing fail-closed behavior preserved)
  - Fix admin path boundary matching: `/minio/administrator` no longer falsely
    matches as admin prefix
  - Normalize Content-Length: 0 for admin GET requests with empty body

  Fixes #3175

* fix(iam): move constant assertion into const block

Fixes clippy::assertions-on-constants warning on
IAM_RETRY_ESCALATION_THRESHOLD assertion.

* fix(iam): address PR review comments

- Replace OnceLock with AtomicU64 sentinel for test isolation;
  add reset_test_failure_counter() for integration tests
- Use u32::try_from() instead of `as u32` narrowing cast in
  compute_backoff_interval
- Rename misleading test; update to verify finalize retry behavior
- Restructure spawn_iam_recovery_task into init-retry and
  finalize-retry phases so transient readiness failures are retried
  instead of leaving the server permanently degraded

* fix(iam): gate test hooks behind debug_assertions

- reset_test_failure_counter() now stores sentinel (u64::MAX) to
  correctly trigger env var re-read on next call
- RUSTFS_TEST_IAM_FAIL_INIT_ATTEMPTS only honored in debug builds
- RUSTFS_TEST_IAM_RETRY_INTERVAL_MS only honored in debug builds

* test(iam): cover deferred bootstrap recovery

Add a dedicated embedded deferred-IAM integration test in a separate test binary to avoid process-global startup collisions.

Strengthen startup IAM recovery coverage with focused unit tests and keep the existing embedded smoke test isolated while carrying the manual test license header update in the same change set.

* fix(startup): tighten deferred IAM recovery path

Adopt follow-up review feedback by silencing misleading app-context warnings after IAM recovery, reusing boundary-aware path prefix checks in the readiness gate, and tying deferred IAM recovery retries to server shutdown tokens.

Keep the deferred IAM embedded integration coverage and startup recovery unit coverage green after the follow-up hardening.

* refactor(startup): simplify IAM recovery task

Collapse the deferred IAM recovery implementation back to a concrete production flow instead of keeping boxed callback seams in the runtime path.

Keep only stable backoff unit coverage in startup_iam and rely on the embedded deferred bootstrap integration test for end-to-end recovery behavior.

* refactor(startup): trim IAM recovery test scaffolding

Keep the concrete deferred IAM recovery path intact while removing bulky test-only async loop scaffolding from startup_iam.

Retain the stable backoff unit checks and rely on the embedded deferred bootstrap integration test for end-to-end recovery coverage.

* fix: apply code review improvements from PR #3188 review

- Simplify RecoveryFuture type alias by removing unnecessary lifetime
- Fix finalize_iam_recovery to return Err if app context unavailable
- Update bootstrap_or_defer_iam_init doc comment to reflect Err case
- Use boundary-aware has_path_prefix for admin path matching in utils.rs
- Add test for adminx boundary rejection in utils.rs and layer.rs
- Improve embedded deferred IAM test with timeout wrapper

* style: merge has_path_prefix import into existing use block

* fix(iam): address final review follow-ups

- fix main startup readiness publication to pass ServiceStateManager correctly
- centralize IAM test env keys in rustfs_config and reuse them in runtime/tests
- keep deferred IAM bootstrap validation aligned with the final review fixes

* fix: isolate listing timeouts from drive health

Keep walk_dir scanner timeouts request-scoped instead of marking local drives faulty.

Add regression coverage for follow-up bucket info, set-level list_path, and system-prefix listings after prior walk timeouts.

* test(iam): gate deferred bootstrap test to debug

Align the deferred IAM embedded integration test with debug-only IAM fault injection hooks so release-profile runs do not assert deferred bootstrap behavior that cannot be triggered.

* test(ecstore): bound prior walk timeout regressions

- set walk_dir stall timeout explicitly in prior-timeout listing tests
- keep the system-prefix follow-up listing scoped to the same base dir
- assert the expected directory entry so the timeout regression test stays fast and stable

* fmt

rustfs/tests/embedded_deferred_iam_test.rs

@@ -0,0 +1,93 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use aws_sdk_s3::config::{Credentials, Region};
+use aws_sdk_s3::{Client, Config};
+use reqwest::StatusCode;
+use rustfs::embedded::{RustFSServerBuilder, find_available_port};
+use rustfs_config::{ENV_TEST_IAM_FAIL_INIT_ATTEMPTS, ENV_TEST_IAM_RETRY_INTERVAL_MS};
+use std::time::Duration;
+use temp_env::async_with_vars;
+
+fn s3_client(endpoint: &str, access_key: &str, secret_key: &str) -> Client {
+    let creds = Credentials::new(access_key, secret_key, None, None, "test");
+    let config = Config::builder()
+        .credentials_provider(creds)
+        .region(Region::new("us-east-1"))
+        .endpoint_url(endpoint)
+        .force_path_style(true)
+        .behavior_version_latest()
+        .build();
+    Client::from_conf(config)
+}
+
+#[cfg(debug_assertions)]
+#[tokio::test]
+async fn test_embedded_server_recovers_after_deferred_iam_bootstrap() {
+    async_with_vars(
+        [
+            (ENV_TEST_IAM_FAIL_INIT_ATTEMPTS, Some("1")),
+            (ENV_TEST_IAM_RETRY_INTERVAL_MS, Some("500")),
+        ],
+        async {
+            let port = find_available_port().expect("find free port");
+            let server = RustFSServerBuilder::new()
+                .address(format!("127.0.0.1:{port}"))
+                .access_key("testaccesskey")
+                .secret_key("testsecretkey")
+                .build()
+                .await
+                .expect("start embedded server with deferred IAM bootstrap");
+
+            let endpoint = server.endpoint();
+            let http = reqwest::Client::new();
+            let ready_url = format!("{endpoint}/health/ready");
+
+            let initial_ready = http
+                .get(&ready_url)
+                .send()
+                .await
+                .expect("readiness probe should respond during deferred bootstrap");
+            assert_eq!(initial_ready.status(), StatusCode::SERVICE_UNAVAILABLE);
+
+            let recovered = tokio::time::timeout(Duration::from_secs(10), async {
+                loop {
+                    tokio::time::sleep(Duration::from_millis(100)).await;
+                    let response = http
+                        .get(&ready_url)
+                        .send()
+                        .await
+                        .expect("readiness probe should keep responding");
+                    if response.status() == StatusCode::OK {
+                        return true;
+                    }
+                }
+            })
+            .await
+            .unwrap_or(false);
+            assert!(recovered, "readiness should recover after deferred IAM bootstrap succeeds");
+
+            let client = s3_client(&endpoint, server.access_key(), server.secret_key());
+            client
+                .create_bucket()
+                .bucket("deferred-bucket")
+                .send()
+                .await
+                .expect("create bucket after readiness recovery");
+
+            server.shutdown().await;
+        },
+    )
+    .await;
+}

rustfs/tests/embedded_test.rs

@@ -1,3 +1,17 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 // Integration test demonstrating the embedded RustFS server API.
 //
 // This test starts a RustFS server in-process and exercises it via the
@@ -39,7 +53,6 @@ async fn test_embedded_server_basic_s3_operations() {
     // 2. Create an S3 client and perform basic operations.
     let client = s3_client(&endpoint, server.access_key(), server.secret_key());
 
-    // Create bucket
     client
         .create_bucket()
         .bucket("test-bucket")
@@ -47,7 +60,6 @@ async fn test_embedded_server_basic_s3_operations() {
         .await
         .expect("create bucket");
 
-    // Put object
     let body = ByteStream::from_static(b"hello rustfs embedded!");
     client
         .put_object()
@@ -58,7 +70,6 @@ async fn test_embedded_server_basic_s3_operations() {
         .await
         .expect("put object");
 
-    // Get object
     let resp = client
         .get_object()
         .bucket("test-bucket")
@@ -70,7 +81,6 @@ async fn test_embedded_server_basic_s3_operations() {
     let data = resp.body.collect().await.expect("read body").into_bytes();
     assert_eq!(data.as_ref(), b"hello rustfs embedded!");
 
-    // List objects
     let list = client
         .list_objects_v2()
         .bucket("test-bucket")
@@ -79,7 +89,6 @@ async fn test_embedded_server_basic_s3_operations() {
         .expect("list objects");
     assert_eq!(list.key_count(), Some(1));
 
-    // Delete object
     client
         .delete_object()
         .bucket("test-bucket")
@@ -88,7 +97,6 @@ async fn test_embedded_server_basic_s3_operations() {
         .await
         .expect("delete object");
 
-    // Delete bucket
     client
         .delete_bucket()
         .bucket("test-bucket")
@@ -96,6 +104,5 @@ async fn test_embedded_server_basic_s3_operations() {
         .await
         .expect("delete bucket");
 
-    // 3. Shut down.
     server.shutdown().await;
 }

rustfs/tests/manual/test_dial9.rs

@@ -1,3 +1,17 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 // Manual Dial9 integration runner.
 //
 // Run with: