mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-18 10:43:15 +00:00
houseme
a08de9229b
* feat(common): add MRF intent channel and Mrf request source (HS-01) Introduce the producer-facing half of the mission repair feed: a global bounded (8192) channel carrying lightweight MrfIntent values from IO error paths, plus the RUSTFS_HEAL_MRF_ENABLE delivery kill-switch and config constants for queue/journal sizing. Delivery is strictly non-blocking (try_send, drop-on-full) so it can sit on decode-failure and partial-write paths without adding latency. HealRequestSource grows a 'mrf' variant so admission accounting can attribute replayed intents. Part of backlog#1865 (option a: wire HealEvent-style intents with a durable retry ledger). Co-Authored-By: heihutu <heihutu@gmail.com> * feat(heal): add MRF queue, durable journal, and intent consumer (HS-01) Consumer half of the mission repair feed: a bounded pending queue (100k intents / 8 MiB dual ceiling, drop-newest on overflow), a durable journal at buckets/.heal/mrf/journal.bin holding the unaccepted pending snapshot, and a consumer task that batches intents off the global channel, translates them into prioritized heal requests (decode failure -> Urgent ECDecode, metadata corruption -> High Metadata, partial write -> Normal object heal), and retries full admissions with a 5s backoff and a 3-attempt ceiling. Durability: every journal record carries its own CRC32 and a format/version header, so a torn tail truncates cleanly at replay; the journal is deleted after a successful replay and when the pending set drains (mirroring MinIO's post-replay list.bin unlink). Losing the last 500 ms flush window is acceptable: replayed duplicates merge via the manager dedup key and read-repair remains the safety net. Metrics: rustfs_heal_mrf_queue_depth/_queue_bytes, _dropped_total {reason}, _replayed_total, _journal_bytes, _journal_fsync_total. The consumer is wired at heal runtime bootstrap right after manager start, honoring RUSTFS_HEAL_MRF_ENABLE (default on, rollback = off). Tests: unit tests for the dual ceiling, record roundtrip, torn-tail truncation, and the priority mapping; integration tests against a real 4-disk ECStore proving channel intents reach the manager queue as Urgent/mrf-attributed requests and journal replay arms intents, drops torn tails, and removes the file. Part of backlog#1865 (option a). Co-Authored-By: heihutu <heihutu@gmail.com> * feat(ecstore,scanner): deliver MRF intents from error paths (HS-01) Wire the three production delivery points, each a single non-blocking try_send next to the existing in-memory heal paths, which stay as the fast path: - read.rs decode-error branch: DecodeFailure intent beside the existing read-repair submit, so an Urgent ECDecode request survives restarts even when the Low-priority read-repair request was dropped or lost. - add_partial: PartialWrite intent, giving partial-write recovery a durable Normal-priority object heal across restarts. - scanner_folder metadata-corruption classification: MetadataCorruption intent beside the existing High-priority scanner heal request. All three are on error paths only: zero cost on healthy IO. Part of backlog#1865 (option a). Co-Authored-By: heihutu <heihutu@gmail.com> * fix: include mrf heal source counts Co-Authored-By: heihutu <heihutu@gmail.com> * fix: keep node heal status wire compatibility Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com>
208 lines
12 KiB
Rust
208 lines
12 KiB
Rust
// Copyright 2024 RustFS Team
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
/// Heal admin config subsystem name.
|
|
pub const HEAL_SUB_SYS: &str = "heal";
|
|
|
|
/// Heal config key setting the scanner-driven periodic deep bitrot scan cycle in seconds.
|
|
pub const HEAL_BITROT_CYCLE: &str = "bitrot_cycle";
|
|
|
|
/// Heal config keys supported by the admin config subsystem.
|
|
pub const HEAL_KEYS: &[&str] = &[HEAL_BITROT_CYCLE];
|
|
|
|
/// Default scanner-driven bitrot scan cycle used by heal/scanner runtime config.
|
|
pub const DEFAULT_HEAL_BITROT_CYCLE_SECS: u64 = 30 * 24 * 60 * 60;
|
|
|
|
/// Environment variable name that enables or disables auto-heal functionality.
|
|
/// - Purpose: Control whether the system automatically performs heal operations.
|
|
/// - Valid values: "true" or "false" (case insensitive).
|
|
/// - Semantics: When set to "true", auto-heal is enabled and the system will automatically attempt to heal detected issues; when set to "false", auto-heal is disabled and healing must be triggered manually.
|
|
/// - Example: `export RUSTFS_HEAL_AUTO_HEAL_ENABLE=true`
|
|
/// - Note: Enabling auto-heal can improve system resilience by automatically addressing issues, but may increase resource usage; evaluate based on your operational requirements.
|
|
pub const ENV_HEAL_AUTO_HEAL_ENABLE: &str = "RUSTFS_HEAL_AUTO_HEAL_ENABLE";
|
|
|
|
/// Environment variable name that specifies the heal queue size.
|
|
///
|
|
/// - Purpose: Set the maximum number of heal requests that can be queued.
|
|
/// - Unit: number of requests (usize).
|
|
/// - Valid values: any positive integer.
|
|
/// - Semantics: When the heal queue reaches this size, new heal requests may be rejected or blocked until space is available; tune according to expected heal workload and system capacity.
|
|
/// - Example: `export RUSTFS_HEAL_QUEUE_SIZE=10000`
|
|
/// - Note: A larger queue size can accommodate bursts of heal requests but may increase memory usage.
|
|
pub const ENV_HEAL_QUEUE_SIZE: &str = "RUSTFS_HEAL_QUEUE_SIZE";
|
|
/// Environment variable name that specifies the heal interval in seconds.
|
|
/// - Purpose: Define the time interval between successive heal operations.
|
|
/// - Unit: seconds (u64).
|
|
/// - Valid values: any positive integer.
|
|
/// - Semantics: This interval controls how frequently the heal manager checks for and processes heal requests; shorter intervals lead to more responsive healing but may increase system load.
|
|
/// - Example: `export RUSTFS_HEAL_INTERVAL_SECS=10`
|
|
/// - Note: Choose an interval that balances healing responsiveness with overall system performance.
|
|
pub const ENV_HEAL_INTERVAL_SECS: &str = "RUSTFS_HEAL_INTERVAL_SECS";
|
|
|
|
/// Environment variable name that specifies the heal task timeout in seconds.
|
|
/// - Purpose: Set the maximum duration allowed for a heal task to complete.
|
|
/// - Unit: seconds (u64).
|
|
/// - Valid values: any positive integer.
|
|
/// - Semantics: If a heal task exceeds this timeout, it may be aborted or retried; tune according to the expected duration of heal operations and system performance characteristics.
|
|
/// - Example: `export RUSTFS_HEAL_TASK_TIMEOUT_SECS=300`
|
|
/// - Note: Setting an appropriate timeout helps prevent long-running heal tasks from impacting system stability.
|
|
pub const ENV_HEAL_TASK_TIMEOUT_SECS: &str = "RUSTFS_HEAL_TASK_TIMEOUT_SECS";
|
|
|
|
/// Environment variable name that specifies the maximum number of concurrent heal operations.
|
|
/// - Purpose: Limit the number of heal operations that can run simultaneously.
|
|
/// - Unit: number of operations (usize).
|
|
/// - Valid values: any positive integer.
|
|
/// - Semantics: This limit helps control resource usage during healing; tune according to system capacity and expected heal workload.
|
|
/// - Example: `export RUSTFS_HEAL_MAX_CONCURRENT_HEALS=4`
|
|
/// - Note: A higher concurrency limit can speed up healing but may lead to resource contention.
|
|
pub const ENV_HEAL_MAX_CONCURRENT_HEALS: &str = "RUSTFS_HEAL_MAX_CONCURRENT_HEALS";
|
|
|
|
/// Environment variable name that specifies the maximum number of concurrent heal operations
|
|
/// allowed for a single erasure set.
|
|
///
|
|
/// - Purpose: Prevent one degraded set from consuming all global heal slots.
|
|
/// - Unit: number of operations (usize).
|
|
/// - Valid values: any positive integer.
|
|
/// - Example: `export RUSTFS_HEAL_MAX_CONCURRENT_PER_SET=1`
|
|
pub const ENV_HEAL_MAX_CONCURRENT_PER_SET: &str = "RUSTFS_HEAL_MAX_CONCURRENT_PER_SET";
|
|
|
|
/// Default value for enabling authentication for heal operations if not specified in the environment variable.
|
|
/// - Value: true (authentication enabled).
|
|
/// - Rationale: Enabling authentication by default enhances security for heal operations.
|
|
/// - Adjustments: Users may disable this feature via the `RUSTFS_HEAL_AUTO_HEAL_ENABLE` environment variable based on their security requirements.
|
|
pub const DEFAULT_HEAL_AUTO_HEAL_ENABLE: bool = true;
|
|
|
|
/// Default heal queue size if not specified in the environment variable.
|
|
///
|
|
/// - Value: 10,000 requests.
|
|
/// - Rationale: This default size balances the need to handle typical heal workloads without excessive memory consumption.
|
|
/// - Adjustments: Users may modify this value via the `RUSTFS_HEAL_QUEUE_SIZE` environment variable based on their specific use cases and system capabilities.
|
|
pub const DEFAULT_HEAL_QUEUE_SIZE: usize = 10_000;
|
|
|
|
/// Default heal interval in seconds if not specified in the environment variable.
|
|
/// - Value: 10 seconds.
|
|
/// - Rationale: This default interval provides a reasonable balance between healing responsiveness and system load for most deployments.
|
|
/// - Adjustments: Users may modify this value via the `RUSTFS_HEAL_INTERVAL_SECS` environment variable based on their specific healing requirements and system performance.
|
|
pub const DEFAULT_HEAL_INTERVAL_SECS: u64 = 10;
|
|
|
|
/// Default heal task timeout in seconds if not specified in the environment variable.
|
|
/// - Value: 300 seconds (5 minutes).
|
|
/// - Rationale: This default timeout allows sufficient time for most heal operations to complete while preventing excessively long-running tasks.
|
|
/// - Adjustments: Users may modify this value via the `RUSTFS_HEAL_TASK_TIMEOUT_SECS` environment variable based on their specific heal operation characteristics and system performance.
|
|
pub const DEFAULT_HEAL_TASK_TIMEOUT_SECS: u64 = 300; // 5 minutes
|
|
|
|
/// Default maximum number of concurrent heal operations if not specified in the environment variable.
|
|
/// - Value: 4 concurrent heal operations.
|
|
/// - Rationale: This default concurrency limit helps balance healing speed with resource usage, preventing system overload.
|
|
/// - Adjustments: Users may modify this value via the `RUSTFS_HEAL_MAX_CONCURRENT_HEALS` environment variable based on their system capacity and expected heal workload.
|
|
pub const DEFAULT_HEAL_MAX_CONCURRENT_HEALS: usize = 4;
|
|
|
|
/// Default maximum number of concurrent heal operations per erasure set.
|
|
///
|
|
/// - Value: 1 concurrent heal operation per set.
|
|
/// - Rationale: Keeps a degraded set from monopolizing the global heal scheduler.
|
|
pub const DEFAULT_HEAL_MAX_CONCURRENT_PER_SET: usize = 1;
|
|
|
|
/// Environment variable that controls whether low-priority heal requests should merge into
|
|
/// an existing queued request with the same deduplication key.
|
|
pub const ENV_HEAL_LOW_PRIORITY_MERGE_ENABLE: &str = "RUSTFS_HEAL_LOW_PRIORITY_MERGE_ENABLE";
|
|
|
|
/// Environment variable that allows low-priority heal requests to be dropped when the queue is full.
|
|
pub const ENV_HEAL_LOW_PRIORITY_DROP_WHEN_FULL: &str = "RUSTFS_HEAL_LOW_PRIORITY_DROP_WHEN_FULL";
|
|
|
|
/// Environment variable that controls concurrent object heals within a single erasure-set page.
|
|
pub const ENV_HEAL_PAGE_OBJECT_CONCURRENCY: &str = "RUSTFS_HEAL_PAGE_OBJECT_CONCURRENCY";
|
|
|
|
/// Environment variable that toggles notify-driven scheduler wakeups.
|
|
pub const ENV_HEAL_EVENT_DRIVEN_SCHEDULER_ENABLE: &str = "RUSTFS_HEAL_EVENT_DRIVEN_SCHEDULER_ENABLE";
|
|
|
|
/// Environment variable that toggles per-set bulkhead scheduling.
|
|
pub const ENV_HEAL_SET_BULKHEAD_ENABLE: &str = "RUSTFS_HEAL_SET_BULKHEAD_ENABLE";
|
|
|
|
/// Environment variable that toggles page-level parallel object healing for erasure-set repair.
|
|
pub const ENV_HEAL_PAGE_PARALLEL_ENABLE: &str = "RUSTFS_HEAL_PAGE_PARALLEL_ENABLE";
|
|
|
|
/// Environment variable that toggles foreground read pressure gating for background heal work.
|
|
pub const ENV_HEAL_MAINLINE_THROTTLE_ENABLE: &str = "RUSTFS_HEAL_MAINLINE_THROTTLE_ENABLE";
|
|
|
|
/// Environment variable that controls the foreground read permit utilization percentage
|
|
/// at which background heal work pauses starting new tasks.
|
|
pub const ENV_HEAL_MAINLINE_READ_UTILIZATION_HIGH_PERCENT: &str = "RUSTFS_HEAL_MAINLINE_READ_UTILIZATION_HIGH_PERCENT";
|
|
|
|
/// Environment variable that controls the foreground write utilization percentage
|
|
/// at which background heal work pauses starting new tasks.
|
|
pub const ENV_HEAL_MAINLINE_WRITE_UTILIZATION_HIGH_PERCENT: &str = "RUSTFS_HEAL_MAINLINE_WRITE_UTILIZATION_HIGH_PERCENT";
|
|
|
|
/// Environment variable that controls how soon the heal scheduler rechecks foreground
|
|
/// pressure after delaying background work.
|
|
pub const ENV_HEAL_MAINLINE_MAX_SLEEP_MS: &str = "RUSTFS_HEAL_MAINLINE_MAX_SLEEP_MS";
|
|
|
|
/// Default behavior is to merge duplicate low-priority requests.
|
|
pub const DEFAULT_HEAL_LOW_PRIORITY_MERGE_ENABLE: bool = true;
|
|
|
|
/// Default behavior is to drop low-priority requests instead of blocking when the queue is full.
|
|
pub const DEFAULT_HEAL_LOW_PRIORITY_DROP_WHEN_FULL: bool = true;
|
|
|
|
/// Default per-page object heal concurrency for erasure-set healing.
|
|
pub const DEFAULT_HEAL_PAGE_OBJECT_CONCURRENCY: usize = 8;
|
|
|
|
/// Default behavior is to keep notify-driven scheduler wakeups enabled.
|
|
pub const DEFAULT_HEAL_EVENT_DRIVEN_SCHEDULER_ENABLE: bool = true;
|
|
|
|
/// Default behavior is to keep per-set bulkhead scheduling enabled.
|
|
pub const DEFAULT_HEAL_SET_BULKHEAD_ENABLE: bool = true;
|
|
|
|
/// Default behavior is to keep erasure-set page parallelism enabled.
|
|
pub const DEFAULT_HEAL_PAGE_PARALLEL_ENABLE: bool = true;
|
|
|
|
/// Default behavior is to pause best-effort heal task starts when foreground reads are saturated.
|
|
pub const DEFAULT_HEAL_MAINLINE_THROTTLE_ENABLE: bool = true;
|
|
|
|
/// Default foreground read permit utilization threshold for pausing best-effort heal task starts.
|
|
pub const DEFAULT_HEAL_MAINLINE_READ_UTILIZATION_HIGH_PERCENT: usize = 80;
|
|
|
|
/// Default foreground write utilization threshold for pausing best-effort heal task starts.
|
|
pub const DEFAULT_HEAL_MAINLINE_WRITE_UTILIZATION_HIGH_PERCENT: usize = 80;
|
|
|
|
/// Default foreground pressure recheck delay for heal scheduler, in milliseconds.
|
|
pub const DEFAULT_HEAL_MAINLINE_MAX_SLEEP_MS: u64 = 250;
|
|
|
|
/// Environment variable that toggles the MRF (mission repair feed) intent
|
|
/// pipeline: error paths deliver repair intents to the heal runtime, and
|
|
/// unconsumed intents are replayed from the durable journal after a restart.
|
|
pub const ENV_HEAL_MRF_ENABLE: &str = "RUSTFS_HEAL_MRF_ENABLE";
|
|
|
|
/// Environment variable for the MRF in-memory queue capacity (intent count).
|
|
pub const ENV_HEAL_MRF_QUEUE_SIZE: &str = "RUSTFS_HEAL_MRF_QUEUE_SIZE";
|
|
|
|
/// Environment variable for the MRF journal byte budget. The journal is
|
|
/// compacted once its on-disk size crosses this bound.
|
|
pub const ENV_HEAL_MRF_JOURNAL_MAX_BYTES: &str = "RUSTFS_HEAL_MRF_JOURNAL_MAX_BYTES";
|
|
|
|
/// Environment variable for the MRF journal replay batch size (intents per
|
|
/// replay push round).
|
|
pub const ENV_HEAL_MRF_REPLAY_BATCH: &str = "RUSTFS_HEAL_MRF_REPLAY_BATCH";
|
|
|
|
/// Default behavior keeps the MRF intent pipeline enabled.
|
|
pub const DEFAULT_HEAL_MRF_ENABLE: bool = true;
|
|
|
|
/// Default MRF queue capacity (matches MinIO's 100k MRF list ceiling).
|
|
pub const DEFAULT_HEAL_MRF_QUEUE_SIZE: usize = 100_000;
|
|
|
|
/// Default MRF journal byte budget (8 MiB), mirroring the channel payload cap.
|
|
pub const DEFAULT_HEAL_MRF_JOURNAL_MAX_BYTES: usize = 8 * 1024 * 1024;
|
|
|
|
/// Default MRF replay batch size.
|
|
pub const DEFAULT_HEAL_MRF_REPLAY_BATCH: usize = 256;
|