Swift container and account metadata handlers cloned the cached
BucketMetadata, set the tagging fields, and called set_bucket_metadata,
which only updates the in-memory cache map. Nothing reached
.metadata.bin, so every Swift metadata POST was lost on restart and
silently overwritten by the next disk-truth reload (a peer
LoadBucketMetadata notification or the 15-minute refresh loop) — while
the client had already been told 2xx.
Route these writes through a new metadata_sys::update_config_with: a
read-modify-write that loads the on-disk metadata and persists the
result under the same write guard metadata_sys::update uses, so the
rewrite merges against disk truth instead of a possibly stale cache and
cannot clobber a concurrent update to another config file. Peers are
notified afterwards, matching the S3 config handlers.
Persisting these writes required hardening the paths that now produce
durable state:
- Account metadata writes validate account ownership. This metadata
holds the account's TempURL signing key, so an unauthenticated write
for someone else's account would have become a durable, cluster-wide
takeover of that account's pre-signed URLs. Reads stay open because
TempURL signature validation runs before credentials exist.
- disable_versioning verifies the container exists. Without it the
metadata loader's "no metadata on disk" default would be persisted,
creating an orphan metadata file and caching a fabricated default as
authoritative.
- Container and account metadata are size- and count-limited, reusing
the Swift limits object metadata already enforces; these tags land in
the bucket metadata file that every later config write rewrites whole.
- A rewrite refuses to run when the persisted tagging config is
unreadable, instead of merging onto an empty set and wiping the
container ACL and versioning tags. It reports 409 naming the remedy.
- Storage errors are logged in full and reported generically, since they
now carry real disk and quorum detail.
The tagging arm of BucketMetadata::update_config also clears the parsed
config, as the lifecycle arm does: parse_all_configs skips empty XML
rather than clearing, so a cleared config kept serving the old tags.
Tagging is serialized with the S3 XML serializer the loader can parse
back, not quick_xml, whose output was never round-trippable.
BucketMetadataSys::get_config lazily fabricated a default BucketMetadata
(object-lock off) for any bucket whose .metadata.bin was ConfigNotFound and
cached it in the map that the map-only, fail-closed metadata_sys::get()
serves. The object-lock batch-delete gate (object_lock_delete_check_required,
backlog#929 / #4297) treats that map as authoritative, so a metadata miss
became a cached "no lock" answer: a versioning peek could poison the cache
and let delete_objects skip the per-object retention/legal-hold stat. The
same fabrication raced make_bucket (lost update overwriting freshly
persisted lock-enabled metadata) and let the 15-minute refresh loop replace
good cached metadata on a transient quorum dip.
Production changes:
- get_config caches only metadata actually read from disk; misses are
recorded in a bounded negative cache (30s TTL, 10k entries, invalidated by
set()) so repeated lookups for metadata-less names cost no extra
namespace-lock + erasure-set fanout (reachable pre-auth via CORS
preflight and per-key in DeleteObjects).
- concurrent_load never lets a fabricated default REPLACE an existing map
entry; startup insert-if-vacant behavior for legacy buckets is preserved.
- delete_objects and new_ns_lock resolve dist-erasure, versioning, and the
object-lock gate from the set's own instance context (backlog#1052)
instead of the ambient facade, so a second in-process instance (or, in
tests, another test's transient DistErasure window) cannot reroute
locking onto an empty dist locker list or answer with the wrong
instance's bucket state.
Test-isolation changes (the bug that surfaced all of the above: the
delete_objects lock-gating test failed deterministically when sharing a
process with the lifecycle env tests):
- The MinIO-migration test builds on an isolated InstanceContext instead of
registering soon-deleted disks in the shared bootstrap registry.
- The cached lifecycle env re-registers its disks on every use, surviving
other serial tests' reset_local_disk_test_state.
- Hermetic SetDisks helpers gain isolated-context variants pinned to plain
erasure; tier-free non-serial test modules use them, guard-based
SetupTypeGuard tests stay on the bootstrap context.
- Three deterministic pin tests (nextest-safe) cover the caching contract,
the delete gate resolution source, and the ns-lock resolution source.
Verification:
- cargo test -p rustfs-ecstore --lib -- --exact <4-test combo from the
report> (previously failing, now green)
- cargo test -p rustfs-ecstore --lib: 3169 passed / 0 failed across
repeated runs; cargo fmt --check and cargo clippy --lib --tests clean
- Adversarial validation (high-risk tier, all seven roles) run per
AGENTS.md; all findings fixed or rebutted with evidence
backlog#1052 S7 — the final piece: full bucket-namespace isolation
between embedded servers in one process.
Server B's requests resolved B's own ECStore (per-server dispatch landed
earlier), but the store's bucket operations still went through ambient
process facades, so both servers effectively operated on the FIRST
server's disks and metadata:
- LocalPeerS3Client::local_disks_for_pools() called all_local_disk()
(the ambient disk registry = the first published store's context), so
list/make/delete/heal bucket scanned and wrote the wrong volumes.
- BucketMetadata::save() persisted through the ambient object handle, so
a second server's bucket metadata landed in the first server's
.rustfs.sys; set/remove/get/created_at all used the ambient metadata
system.
Now the whole chain is bound to the owning store's InstanceContext:
- S3PeerSys/LocalPeerS3Client gain *_with_instance_ctx constructors and
operate on that context's registered disks; ECStore::new (and the test
store builder) pass the store's context. The legacy constructors keep
the bootstrap default.
- BucketMetadata::save_with_store persists through an explicit store;
BucketMetadataSys::persist_and_set uses the system's own api handle.
- metadata_sys gains instance-scoped variants (get_in / created_at_in /
set_bucket_metadata_in / remove_bucket_metadata_in) that resolve the
context's metadata system and fall back to the ambient default before
the instance cell is initialized (early startup, unchanged behavior).
- The store's bucket handlers (make/get_info/list/delete + the
table-bucket delete guard and emptiness check) use the per-context
variants and this instance's disks.
Acceptance (e2e): two embedded servers with different credentials are now
isolated end to end — each authenticates only its own key, neither sees
the other's buckets or objects, and both data planes stay intact. The
embedded module doc drops the shared-IAM caveat.
579 ecstore bucket/metadata/peer regressions plus the embedded basic and
deferred-IAM e2e stay green.
perf(ecstore): per-bucket durability tier overrides (HP-5 phase 2)
Let a bucket override the process-wide RUSTFS_DURABILITY_MODE with its own
strict/relaxed/none tier, stored as a durability.json extension entry in the
bucket metadata file and resolved at commit points via effective_durability.
System-critical buckets (.rustfs.sys, .minio.sys) can never carry an override
and stay pinned to strict; the legacy full-off switch keeps its historical
semantics and per-bucket overrides do not apply under it. Overrides are
published and cleared through the existing bucket metadata cache-invalidation
path, and an admin GET/PUT handler exposes the configuration. Default behavior
is unchanged: with no override a bucket follows the global mode, which defaults
to strict and stays byte-for-byte identical to before.
Refs: https://github.com/rustfs/backlog/issues/938, https://github.com/rustfs/backlog/issues/936
Co-authored-by: heihutu <heihutu@gmail.com>
test(interop): real-MinIO read + migration parity, Phase 1/2 (backlog#580)
Capture authentic on-disk fixtures from MinIO RELEASE.2025-07-23 (a bucket with
versioning, object-lock, lifecycle, tagging, quota, a public policy, SSE-S3
encryption, a webhook notification target, and a replication rule, plus inline /
versioned / multipart objects and a delete marker) and prove RustFS reads and
migrates them losslessly:
- filemeta parses_real_minio_object_xlmeta: small inline, two-object-version +
delete marker, and multipart object xl.meta parse to the expected FileInfo.
- ecstore parses_real_minio_bucket_metadata_blob_without_loss: the MinIO
.metadata.bin msgpack decodes via the PascalCase field names and
parse_all_configs loads all ten config types present (policy, lifecycle incl.
<ExpiryUpdatedAt>, object-lock, versioning, tagging, quota, notification,
encryption/SSE-S3, replication incl. DeleteMarkerReplication /
ExistingObjectReplication) without loss.
- ecstore reads_minio_inline_bucket_metadata_via_bitrot: MinIO inlines an object
body as [HighwayHash256 32B][body]; RustFS's BitrotReader with HighwayHash256S
verifies and yields the exact blob (the "inline_data 前缀不同" is that prefix).
- ecstore migrates_real_minio_bucket_metadata_end_to_end: on a throwaway 4-drive
local ECStore, a real MinIO .metadata.bin seeded under .minio.sys is migrated
into .rustfs.sys byte-identically for every config, exercising the Phase 2
source adapter (MIGRATING_META_BUCKET = ".minio.sys") through the object layer.
All four run as ordinary crate tests (nextest CI). Phase 4 (MinIO re-reading a
RustFS drive) is documented as out of scope for one-way migration.
Refs rustfs/backlog#580
This commit introduces a significant reorganization of the project structure to improve maintainability and clarity.
Key changes include:
- Adjusted the directory layout for a more logical module organization.
- Removed unused crate dependencies, reducing the overall project size and potentially speeding up build times.
- Updated import paths and configuration files to reflect the structural changes.
Zhengchao An