feat(ecstore): run bucket operations on the store's own instance context (#4642)

backlog#1052 S7 — the final piece: full bucket-namespace isolation
between embedded servers in one process.

Server B's requests resolved B's own ECStore (per-server dispatch landed
earlier), but the store's bucket operations still went through ambient
process facades, so both servers effectively operated on the FIRST
server's disks and metadata:

- LocalPeerS3Client::local_disks_for_pools() called all_local_disk()
  (the ambient disk registry = the first published store's context), so
  list/make/delete/heal bucket scanned and wrote the wrong volumes.
- BucketMetadata::save() persisted through the ambient object handle, so
  a second server's bucket metadata landed in the first server's
  .rustfs.sys; set/remove/get/created_at all used the ambient metadata
  system.

Now the whole chain is bound to the owning store's InstanceContext:

- S3PeerSys/LocalPeerS3Client gain *_with_instance_ctx constructors and
  operate on that context's registered disks; ECStore::new (and the test
  store builder) pass the store's context. The legacy constructors keep
  the bootstrap default.
- BucketMetadata::save_with_store persists through an explicit store;
  BucketMetadataSys::persist_and_set uses the system's own api handle.
- metadata_sys gains instance-scoped variants (get_in / created_at_in /
  set_bucket_metadata_in / remove_bucket_metadata_in) that resolve the
  context's metadata system and fall back to the ambient default before
  the instance cell is initialized (early startup, unchanged behavior).
- The store's bucket handlers (make/get_info/list/delete + the
  table-bucket delete guard and emptiness check) use the per-context
  variants and this instance's disks.

Acceptance (e2e): two embedded servers with different credentials are now
isolated end to end — each authenticates only its own key, neither sees
the other's buckets or objects, and both data planes stay intact. The
embedded module doc drops the shared-IAM caveat.

579 ecstore bucket/metadata/peer regressions plus the embedded basic and
deferred-IAM e2e stay green.
This commit is contained in:
Zhengchao An
2026-07-10 10:52:51 +08:00
committed by GitHub
parent e6e4aef45b
commit dc3099bf0f
9 changed files with 192 additions and 44 deletions
+79 -12
View File
@@ -105,19 +105,13 @@ async fn two_embedded_servers_start_and_shutdown_independently() {
server_a.shutdown().await;
}
// backlog#1052 auth acceptance: two embedded servers with *different*
// credentials each authenticate against their own root identity. Server B
// accepts its own access key and rejects server A's. This exercises the
// per-server auth path (each request resolves its own AppContext for
// credential validation).
//
// NOTE: full bucket-namespace isolation is a separate, deeper follow-up: the
// ecstore data plane still resolves some lower-level reads (peer/disk/bucket
// metadata) through the process-global object handle, so the two servers do
// not yet present independent bucket listings even though each holds its own
// store object. That isolation is the remaining work on #1052.
// backlog#1052 full acceptance: two embedded servers with *different*
// credentials are isolated end to end — auth (each accepts its own key and
// rejects the other's) AND data plane (each server's buckets/objects are
// invisible to the other; each lists/creates/deletes only on its own disks
// and bucket-metadata system).
#[tokio::test]
async fn two_embedded_servers_authenticate_with_their_own_credentials() {
async fn two_embedded_servers_isolate_auth_and_data_planes() {
let port_a = match find_available_port() {
Ok(port) => port,
Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return,
@@ -176,6 +170,79 @@ async fn two_embedded_servers_authenticate_with_their_own_credentials() {
.await
.expect("server A must authenticate with its own credentials");
// ---- Data-plane isolation (backlog#1052 S7) ----
// Server A owns a bucket + object.
client_a
.create_bucket()
.bucket("only-on-a")
.send()
.await
.expect("server A creates its bucket");
client_a
.put_object()
.bucket("only-on-a")
.key("marker.txt")
.body(ByteStream::from_static(b"belongs to A"))
.send()
.await
.expect("server A writes its object");
// Server B's listing does not contain server A's bucket.
let b_buckets: Vec<_> = client_b
.list_buckets()
.send()
.await
.expect("server B lists buckets")
.buckets()
.iter()
.flat_map(|bucket| bucket.name.clone())
.collect();
assert!(
!b_buckets.contains(&"only-on-a".to_string()),
"server B must not see server A's bucket; saw {b_buckets:?}"
);
// Server B cannot resolve server A's object either.
let cross_head = client_b.head_object().bucket("only-on-a").key("marker.txt").send().await;
assert!(cross_head.is_err(), "server B must not resolve server A's object; got {cross_head:?}");
// Server B's own bucket is invisible to server A.
client_b
.create_bucket()
.bucket("only-on-b")
.send()
.await
.expect("server B creates its bucket");
let a_buckets: Vec<_> = client_a
.list_buckets()
.send()
.await
.expect("server A lists buckets")
.buckets()
.iter()
.flat_map(|bucket| bucket.name.clone())
.collect();
assert!(
a_buckets.contains(&"only-on-a".to_string()),
"server A must keep seeing its own bucket; saw {a_buckets:?}"
);
assert!(
!a_buckets.contains(&"only-on-b".to_string()),
"server A must not see server B's bucket; saw {a_buckets:?}"
);
// Server A's data plane is intact.
let a_get = client_a
.get_object()
.bucket("only-on-a")
.key("marker.txt")
.send()
.await
.expect("server A serves its own object");
let a_data = a_get.body.collect().await.expect("read A body").into_bytes();
assert_eq!(a_data.as_ref(), b"belongs to A");
server_a.shutdown().await;
server_b.shutdown().await;
}