feat(ecstore): run bucket operations on the store's own instance context (#4642)

backlog#1052 S7 — the final piece: full bucket-namespace isolation
between embedded servers in one process.

Server B's requests resolved B's own ECStore (per-server dispatch landed
earlier), but the store's bucket operations still went through ambient
process facades, so both servers effectively operated on the FIRST
server's disks and metadata:

- LocalPeerS3Client::local_disks_for_pools() called all_local_disk()
  (the ambient disk registry = the first published store's context), so
  list/make/delete/heal bucket scanned and wrote the wrong volumes.
- BucketMetadata::save() persisted through the ambient object handle, so
  a second server's bucket metadata landed in the first server's
  .rustfs.sys; set/remove/get/created_at all used the ambient metadata
  system.

Now the whole chain is bound to the owning store's InstanceContext:

- S3PeerSys/LocalPeerS3Client gain *_with_instance_ctx constructors and
  operate on that context's registered disks; ECStore::new (and the test
  store builder) pass the store's context. The legacy constructors keep
  the bootstrap default.
- BucketMetadata::save_with_store persists through an explicit store;
  BucketMetadataSys::persist_and_set uses the system's own api handle.
- metadata_sys gains instance-scoped variants (get_in / created_at_in /
  set_bucket_metadata_in / remove_bucket_metadata_in) that resolve the
  context's metadata system and fall back to the ambient default before
  the instance cell is initialized (early startup, unchanged behavior).
- The store's bucket handlers (make/get_info/list/delete + the
  table-bucket delete guard and emptiness check) use the per-context
  variants and this instance's disks.

Acceptance (e2e): two embedded servers with different credentials are now
isolated end to end — each authenticates only its own key, neither sees
the other's buckets or objects, and both data planes stay intact. The
embedded module doc drops the shared-IAM caveat.

579 ecstore bucket/metadata/peer regressions plus the embedded basic and
deferred-IAM e2e stay green.

rustfs/src/embedded.rs

@@ -40,15 +40,13 @@
 //! }
 //! ```
 //!
-//! # Multi-instance status
+//! # Multi-instance support
 //!
-//! Multiple `RustFSServer`s may now coexist in one process on different ports
-//! and volumes (backlog#1052). Each server's storage layer, request-path
-//! dispatch, and app subsystem instances stay isolated. IAM and root-credential
-//! validation still share a process-wide domain, so a second server whose
-//! credentials differ from the first cannot authenticate S3 clients today;
-//! wiring authentication per-server is tracked as a follow-up on the same
-//! issue.
+//! Multiple `RustFSServer`s may coexist in one process on different ports and
+//! volumes (backlog#1052). Each server is fully isolated: its own storage
+//! layer, request-path dispatch, root credentials, IAM domain, and bucket
+//! namespace. Two servers with different credentials each authenticate their
+//! own clients, and neither can see the other's buckets or objects.
 
 use crate::server::ShutdownHandle;
 use crate::startup_embedded::{EmbeddedStartedServer, EmbeddedStartupArgs, EmbeddedStartupError, run_embedded_startup};

rustfs/tests/embedded_multi_instance_test.rs

@@ -105,19 +105,13 @@ async fn two_embedded_servers_start_and_shutdown_independently() {
     server_a.shutdown().await;
 }
 
-// backlog#1052 auth acceptance: two embedded servers with *different*
-// credentials each authenticate against their own root identity. Server B
-// accepts its own access key and rejects server A's. This exercises the
-// per-server auth path (each request resolves its own AppContext for
-// credential validation).
-//
-// NOTE: full bucket-namespace isolation is a separate, deeper follow-up: the
-// ecstore data plane still resolves some lower-level reads (peer/disk/bucket
-// metadata) through the process-global object handle, so the two servers do
-// not yet present independent bucket listings even though each holds its own
-// store object. That isolation is the remaining work on #1052.
+// backlog#1052 full acceptance: two embedded servers with *different*
+// credentials are isolated end to end — auth (each accepts its own key and
+// rejects the other's) AND data plane (each server's buckets/objects are
+// invisible to the other; each lists/creates/deletes only on its own disks
+// and bucket-metadata system).
 #[tokio::test]
-async fn two_embedded_servers_authenticate_with_their_own_credentials() {
+async fn two_embedded_servers_isolate_auth_and_data_planes() {
     let port_a = match find_available_port() {
         Ok(port) => port,
         Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return,
@@ -176,6 +170,79 @@ async fn two_embedded_servers_authenticate_with_their_own_credentials() {
         .await
         .expect("server A must authenticate with its own credentials");
 
+    // ---- Data-plane isolation (backlog#1052 S7) ----
+
+    // Server A owns a bucket + object.
+    client_a
+        .create_bucket()
+        .bucket("only-on-a")
+        .send()
+        .await
+        .expect("server A creates its bucket");
+    client_a
+        .put_object()
+        .bucket("only-on-a")
+        .key("marker.txt")
+        .body(ByteStream::from_static(b"belongs to A"))
+        .send()
+        .await
+        .expect("server A writes its object");
+
+    // Server B's listing does not contain server A's bucket.
+    let b_buckets: Vec<_> = client_b
+        .list_buckets()
+        .send()
+        .await
+        .expect("server B lists buckets")
+        .buckets()
+        .iter()
+        .flat_map(|bucket| bucket.name.clone())
+        .collect();
+    assert!(
+        !b_buckets.contains(&"only-on-a".to_string()),
+        "server B must not see server A's bucket; saw {b_buckets:?}"
+    );
+
+    // Server B cannot resolve server A's object either.
+    let cross_head = client_b.head_object().bucket("only-on-a").key("marker.txt").send().await;
+    assert!(cross_head.is_err(), "server B must not resolve server A's object; got {cross_head:?}");
+
+    // Server B's own bucket is invisible to server A.
+    client_b
+        .create_bucket()
+        .bucket("only-on-b")
+        .send()
+        .await
+        .expect("server B creates its bucket");
+    let a_buckets: Vec<_> = client_a
+        .list_buckets()
+        .send()
+        .await
+        .expect("server A lists buckets")
+        .buckets()
+        .iter()
+        .flat_map(|bucket| bucket.name.clone())
+        .collect();
+    assert!(
+        a_buckets.contains(&"only-on-a".to_string()),
+        "server A must keep seeing its own bucket; saw {a_buckets:?}"
+    );
+    assert!(
+        !a_buckets.contains(&"only-on-b".to_string()),
+        "server A must not see server B's bucket; saw {a_buckets:?}"
+    );
+
+    // Server A's data plane is intact.
+    let a_get = client_a
+        .get_object()
+        .bucket("only-on-a")
+        .key("marker.txt")
+        .send()
+        .await
+        .expect("server A serves its own object");
+    let a_data = a_get.body.collect().await.expect("read A body").into_bytes();
+    assert_eq!(a_data.as_ref(), b"belongs to A");
+
     server_a.shutdown().await;
     server_b.shutdown().await;
 }