mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-12 23:52:45 +00:00
cppudge
99ae115731
* fix(ecstore): keep object deletes running when the caller is dropped DeleteObject and DeleteObjects commit on every disk and then run a separate cleanup pass, both inside the request future. When the client disconnects, hyper drops that future and the delete stops between the two passes: xl.meta is gone, but the rollback backup and the marked data dir stay on disk, hidden from listings and never reclaimed (#6898). Run the ECStore delete entry points on a detached task tracked by the instance context. The task owns the object locks, both disk passes and any rollback, so dropping the caller only stops the wait. Server shutdown waits up to 30s for detached deletes after the HTTP drain. Two smaller leaks of the same kind: a cancelled data-dir delete no longer leaves its snapshot-lease entry marked as deleting forever, and DiskStore::delete_versions releases its waiting slot through a guard. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * fix(ecstore): finish committed delete residue on repeated deletes Residue that an older build stranded (#6898) has no xl.meta, so deleting the key again reports it missing and leaves everything in place. On a disk, a delete-data.<T> marker in one of the object's data dirs proves that transaction T removed the metadata there and committed: the data it marked can no longer be read, healed or rolled back on that disk. When delete_versions or delete_version finds no xl.meta, remove what the cleanup pass of every transaction proven that way would have removed, under the object's metadata lease. The rollback backup goes before the marked data dirs, so an attempt cut short still leaves the marker for the next one. Reserved markers, rollback backups without a committed marker and directories that also hold xl.meta or other keys are left alone. The disk result stays FileNotFound; the extra cost is one directory read per missing key inside the disk call that already runs, with no new RPC, and the read stops at the first child that is not a data dir. Finished transactions are counted in rustfs_disk_delete_residue_finished_total instead of logged per object. A single DeleteObject in an unversioned bucket answers a missing key from the metadata read quorum without reaching the disks, so existing residue is reclaimed by DeleteObjects or by the empty-listing purge. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * fix(ecstore): purge the rollback backup of a committed delete on listing After an empty listing, the orphan purge (#7955, #8261) already removes data dirs that carry a committed delete-data.<T> marker, but it treated the <object>/<T>/xl.meta.bkp rollback backup of the same delete as data and blocked the whole object directory and its ancestors. Residue left by an abandoned delete (#6898) therefore survived the listing, and DeleteBucket kept failing with BucketNotEmpty. Treat a UUID dir that holds only xl.meta.bkp as purgeable when a purgeable data dir of the same object on the same disk carries the committed marker of that transaction. A backup without such a marker still blocks as before, since it may be needed to roll the delete back. The other purge gates are unchanged: every disk of the set must be online, and the backoff applies. xl.meta or unmarked data on any disk blocks that directory and its ancestors on every disk. Like a marked data dir before, a backup below such a directory is still removed from a disk that holds no xl.meta there and proves the transaction with its own marker. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * fix(ecstore): move caller-held delete locks into the detached delete A force (recursive) delete takes the bucket lifecycle write lock in the S3 handler and passes only its fence down, so the store does not take that lock again. Since the delete runs on a detached task, dropping the request released the lock while the delete kept running: a PUT under the same prefix could succeed and then be removed by the old delete. Lifecycle expiry had the same shape with its table-bucket publication read lock, of which only the lock-lost signal reaches the delete. Add ECStore::delete_object_with_tier_delete_journal_and_guards. It moves the caller's guards into the detached task and releases them after the delete and its listing bookkeeping. The force-delete handler and lifecycle expiry hand their guards over; DeleteObjects holds none. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * fix(startup): keep unclean-shutdown markers while deletes still run Shutdown waits up to 30s for detached deletes, but it cleared the unclean-shutdown markers even when that wait timed out. A delete then cut short by the runtime teardown went unnoticed: the next startup skipped the erasure-set heal that the marker triggers. Clear the markers only when the heal handoff completed and every detached mutation finished, and log that they are kept otherwise. Wait after the FTP, FTPS, SFTP and WebDAV servers have stopped as well, since their deletes take the same detached path. Signed-off-by: Petr Slavyagin <disident47@gmail.com> * test(ecstore): simulate tier-delete crashes below the detached delete The chunk crash-boundary tests model a crash by aborting the task that runs delete_object_with_tier_delete_journal while it is paused at a TierDeleteChunkTestBarrier. That entry point now runs the delete on a detached task, so aborting the caller no longer stops it: the delete resumed on the old store once the barrier was dropped, and building the restarted store on the same disks tripped the tier-delete journal proof assertion (tier_delete_chunk_crash_boundaries_resume_without_skipping_sources failed in CI). Run the delete through handle_delete_object_with_journal, the body below the detached task, so the abort stops it at the barrier as before. tier_delete_chunk_missing_child_with_journals_fails_closed used the same pattern; its resumed delete only failed fast because the tier rejects removes, so it gets the same change. Signed-off-by: Petr Slavyagin <disident47@gmail.com> --------- Signed-off-by: Petr Slavyagin <disident47@gmail.com>
7.4 KiB
7.4 KiB
Global State Inventory
Use this when: you meet a GLOBAL_* static or an OnceLock and need to know whether it is a runtime ownership handle (reach it through a boundary), an owner-local static (leave it inside its module), or process-global by design.
Source of truth: crates/ecstore/src/api/mod.rs (the pub mod runtime and pub mod global re-export lists), crates/ecstore/src/runtime/global.rs, crates/ecstore/src/runtime/sources.rs, and the statics themselves. Boundary rules are in global-state-crate-split-plan.md.
Global State Classification
| Category | Rule | Representative owners |
|---|---|---|
| Process-global | Process identity, metrics registries, lock manager, audit guard, TLS material, or other state intentionally one per process. | GLOBAL_LOCK_MANAGER (crates/lock), GLOBAL_CONN_MAP (crates/common), GLOBAL_RUSTFS_RPC_SECRET (crates/credentials), AUDIT_SYSTEM (crates/audit), crates/io-metrics, crates/obs, crates/tls-runtime |
| Runtime migration target | Mutable runtime state describing the active object store, endpoints, local disks, lifecycle, replication, notification, config, or background controllers. | crates/ecstore/src/runtime/global.rs, crates/ecstore/src/runtime/sources.rs, rustfs/src/app/context/ |
| Owner-local compatibility | Adapters allowed to read globals while callers migrate to AppContext-first or owner-local runtime-source APIs. | rustfs/src/*/runtime_sources.rs, rustfs/src/*/storage_api.rs, crates/*/storage_api.rs |
| Owner-local static | A static private to one module and reached only through that module's functions: caches, single-run guards, admission locks, module toggles. | The RustFS inventory below |
| Test or fixture state | Static setup that amortizes expensive ECStore setup or isolates harness state. | rustfs/src/app/*_test.rs, crates/scanner/tests/, crates/test-utils/src/ecstore_test_compat.rs |
| Cache or constant | Regexes, metrics descriptors, defaults, KVS registrations, headers, path constants. | crates/config, crates/obs/src/metrics, crates/utils |
Runtime Migration Inventory
Runtime ownership handles that exist today. Reads go through rustfs_ecstore::api::runtime, bootstrap writes go through rustfs_ecstore::api::global, and RustFS code reaches both only from rustfs/src/storage/storage_api.rs and the AppContext resolvers.
Handle (rustfs_ecstore::api::runtime) |
Backing state | Stance |
|---|---|---|
object_store_handle |
GLOBAL_OBJECT_API, GLOBAL_OBJECT_STORE_RESOLVER (crates/ecstore/src/runtime/global.rs); the resolver is published from the AppContext owner path |
Do not migrate first: tied to storage startup, IAM-after-storage AppContext publication, and data-plane resolver compatibility. |
endpoint_pools, setup_is_erasure, setup_is_dist_erasure, setup_is_erasure_sd, first_cluster_node_is_local |
GLOBAL_ENDPOINTS and setup-type state (crates/ecstore/src/runtime/global.rs) |
Move endpoint ownership only after readiness and quorum behavior have explicit coverage. |
local_disk_map_read |
Local disk map and set-drive state (crates/ecstore/src/runtime/sources.rs) |
Preserve disk lookup, remote/local classification, and test reset hooks. |
expiry_state_handle, transition_state_handle |
Lifecycle expiry and transition state, GLOBAL_LIFECYCLE_SYS (crates/ecstore/src/runtime/global.rs) |
Lifecycle owner helpers and the AppContext ExpiryStateInterface (rustfs/src/app/context/interfaces.rs) are the caller boundary; the scanner still reads expiry_state_handle until it gets an injected provider. |
global_tier_config_mgr |
Tier config manager | Reads and reloads stay behind this helper. |
bucket_monitor |
Replication bandwidth monitor | Replication pool/stat handles are projected into RustFS wrapper types at the storage boundary. |
global_lock_client, global_lock_clients |
GLOBAL_LOCAL_LOCK_CLIENT, GLOBAL_LOCK_CLIENTS (crates/ecstore/src/runtime/global.rs) |
Preserve lock quorum and client selection; the process-level GLOBAL_LOCK_MANAGER stays separate. |
boot_time, deployment_id, region, rustfs_port |
GLOBAL_BOOT_TIME, deployment id, region, and port state (crates/ecstore/src/runtime/global.rs) |
Scalar writes remain behind the api::global setters (set_global_endpoints, set_global_region, set_global_rustfs_port, set_object_store_resolver, shutdown_background_services, update_erasure_type, wait_for_detached_mutations). |
Owner-helper handles outside the runtime-source list stay inside their owner and are reached through owner functions: GLOBAL_EVENT_NOTIFIER (crates/ecstore/src/runtime/global.rs); GLOBAL_NOTIFICATION_SYS, EVENT_DISPATCH_HOOK, GLOBAL_PROCESSORS, INTERNODE_DATA_TRANSPORT, GLOBAL_BUCKET_TARGET_SYS, GLOBAL_CONFIG_SYS, GLOBAL_STORAGE_CLASS, WORKLOAD_ADMISSION_SNAPSHOT_PROVIDER (ECStore owner modules); GLOBAL_SERVER_CONFIG (crates/config/src/server_config.rs); GLOBAL_HEAL_RUNTIME, GLOBAL_AHM_SERVICES_CANCEL_TOKEN (crates/heal/src/lib.rs); GLOBAL_KMS_SERVICE_MANAGER (crates/kms/src/service_manager.rs); GLOBAL_CAPACITY_MANAGER (crates/object-capacity/src/capacity_manager.rs); APP_CONTEXT_SINGLETON (rustfs/src/app/context/global.rs).
Regenerate:
rg -n -A4 'pub use crate::runtime::(sources|global)::' crates/ecstore/src/api/mod.rs
rg -n --glob '*.rs' 'static (ref )?GLOBAL_[A-Z_]+' crates rustfs/src
RustFS Owner-Local Static Inventory
RustFS-side statics that matter architecturally because other modules are tempted to reach them. They stay private to their owner module; callers use the owner's functions.
| Static | Owner | Stance |
|---|---|---|
KEYSTONE_AUTH, KEYSTONE_MAPPER, KEYSTONE_CONFIG |
rustfs/src/auth_keystone.rs |
Keystone provider, mapper, and config stay private to the Keystone owner. |
DEADLOCK_DETECTOR |
rustfs/src/storage/deadlock_detector.rs |
Detector lifecycle stays private to the storage deadlock detector. |
CONCURRENCY_MANAGER |
rustfs/src/storage/concurrency/manager.rs |
Storage concurrency scheduler state stays inside the concurrency owner. |
GLOBAL_KMS_DEK_PROVIDER, GLOBAL_SSE_DEK_PROVIDER |
rustfs/src/storage/sse.rs |
DEK provider caches stay private to the SSE owner. |
ECSTORE_EVENT_DISPATCH_HOOK |
rustfs/src/server/event.rs |
Event bridge registration goes through the storage facade. |
AUDIT_MODULE_ENABLED, NOTIFY_MODULE_ENABLED |
rustfs/src/module_switches.rs |
Module toggles are read through module-switch helpers; MODULE_SWITCH_RMW_LOCK (rustfs/src/server/module_switch.rs) serializes persisted updates. |
RUNTIME_CONFIG_RELOAD_MUTEX |
rustfs/src/admin/service/config.rs |
Serializes dynamic config reload fanout. |
EMBEDDED_RUNTIME_OWNERS |
rustfs/src/startup_shutdown.rs |
Embedded runtime owner handles used for shutdown ordering. |
SERVICE_FROZEN |
rustfs/src/admin/handlers/system.rs |
Service freeze flag stays behind the system admin handler. |
RECONCILER |
rustfs/src/site_replication_reconcile.rs |
Site-replication reconciler singleton. |
CONSOLE_CONFIG |
rustfs/src/admin/console.rs |
Console bootstrap config. |
LICENSE_STATE, LICENSE_VERIFIER |
rustfs/src/license.rs |
License state and verifier stay behind license helpers. |
Regenerate the full list (long, mostly caches and test hooks):
rg -n '^\s*(pub(\(crate\))? )?static [A-Z_]+' rustfs/src