mirror of
https://github.com/rustfs/rustfs.git
synced 2026-09-06 12:09:12 +00:00
feat(ecstore): add on-demand migration runtime OnDemandMigrationSys (#7074)
* feat(ecstore): add on-demand migration bucket config model Introduce OnDemandMigrationConfig (deny_unknown_fields, version 1) with typed validation, credential redaction, a secret-free Debug impl, and the OnceLock publish hook the runtime registers into. Exported through the api facade. * feat(ecstore): persist on-demand migration config in bucket metadata Store the config as a RustFS extension entry (on-demand-migration.json) with its update time in .metadata.bin, add the typed BucketMetadataSys accessor, and publish the config through the hook on every cache-install path alongside the durability sync. * refactor(ecstore): extract shared remote S3 client builder Move the aws_sdk_s3 client construction out of bucket_target_sys into bucket/remote_s3_client.rs: endpoint assembly, credential provider, path-style selection, custom CA / skip-TLS transports and the outbound SSRF gate now build from a neutral RemoteS3EndpointSpec so replication targets and the upcoming on-demand migration source client share one policy. Replication builds its client through From<&BucketTarget>; the gate keeps its relaxed semantics (private allowed, loopback only behind RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) verbatim. The builder also gains optional connect/read timeouts and a User-Agent suffix interceptor, both unset for replication. Refs rustfs/backlog#2149 * feat(ecstore): add on-demand migration SourceClient Add bucket/on_demand_migration/source_client.rs on top of the shared remote S3 builder: HEAD, ranged streaming GET, ListObjectsV2 with source-prefix mapping, GetObjectTagging and an admin probe. Every request carries the x-rustfs-/x-minio-source-proxy-request anti-loop markers and a RustFS-OnDemandMigration/<version> User-Agent suffix; SSE-C source objects are rejected as unsupported. SourceError classifies SDK failures (not found, access denied, throttled, timeout, connect, server error) with retryability and a stable metrics label. Debug output redacts credentials. Refs rustfs/backlog#2149 * docs(operations): point outbound policy at shared remote S3 client builder * chore: integrate ODM-01 and ODM-02 as B1 base (fix facade merge) * feat(ecstore): add on-demand migration runtime OnDemandMigrationSys Per-node runtime for On-Demand Migration (rustfs/backlog#2152): turns each bucket's persisted config into a live SourceClient guarded by a three-state circuit breaker, a TTL negative cache, per-key singleflight, a pull concurrency semaphore and lock-free counters with a serializable snapshot. - sys.rs: OnceLock singleton; `apply` installs/rebuilds/removes bucket state (config compared by value, counters preserved across rebuilds, old cancellation token fired); `publish` is the metadata publish-hook entry (sync removal, spawned install, generation-ordered so a slow older install cannot overwrite a newer one); `resolve(bucket, key)` judges module switch, bucket state, prefix filter, client availability, negative cache, breaker. - breaker.rs: Closed/Open/HalfOpen with fixed constants (5 failures / 30 s window / 30 s open / 1 probe); NotFound resets, AccessDenied is neutral. - negative_cache.rs: moka sync cache keyed by local key, ttl=0 disables. - stats.rs: requests_total{op,outcome}, pulled_bytes/objects, pull_failures, inflight/queue gauges, log-bucket latency histogram, last_source_error; snake_case snapshot pinned by a golden JSON test. - Anonymous sources surface as a typed `OdmStateError::AnonymousUnsupported` until the shared client builder gains an anonymous mode. - rustfs: `RUSTFS_ON_DEMAND_MIGRATION_ENABLED` module switch (default false) published to module_switches and injected into ecstore before bucket metadata loads; hook registered at the same point.
This commit is contained in:
@@ -146,6 +146,14 @@ pub mod bucket {
|
||||
}
|
||||
|
||||
pub mod on_demand_migration {
|
||||
pub use crate::bucket::on_demand_migration::{
|
||||
ApplyOutcome, BREAKER_FAILURE_THRESHOLD, BREAKER_FAILURE_WINDOW, BREAKER_HALF_OPEN_MAX_PROBES, BREAKER_OPEN_DURATION,
|
||||
Breaker, BreakerState, BreakerTransition, BreakerVerdict, BucketOdmState, GLOBAL_ON_DEMAND_MIGRATION_SYS, GaugeGuard,
|
||||
LastSourceError, LatencyBucketSnapshot, NEGATIVE_CACHE_MAX_ENTRIES, NegativeCache, OdmBucketSnapshot, OdmLookup,
|
||||
OdmOp, OdmOutcome, OdmStateError, OdmStats, OdmStatsSnapshot, OnDemandMigrationSys, PullError, PullFailureReason,
|
||||
PullFollower, PullLeader, PullOutcome, PullPath, PullResult, PullSlot, SOURCE_LATENCY_BUCKET_BOUNDS_MS,
|
||||
SourceLatencySnapshot, source_client_spec,
|
||||
};
|
||||
pub use crate::bucket::on_demand_migration::{
|
||||
ConfigPublishHook, FilterConfig, HeadPolicy, ON_DEMAND_MIGRATION_CONFIG_HOOK, ON_DEMAND_MIGRATION_CONFIG_VERSION,
|
||||
OnDemandMigrationConfig, OnDemandMigrationConfigError, PathStyle, PolicyConfig, Provider, RangeGetPolicy,
|
||||
|
||||
Reference in New Issue
Block a user