feat(fips): route internode RPC HMAC through aws-lc-rs FIPS (#9137)

* feat(fips): route internode RPC HMAC through aws-lc-rs FIPS

* Revise KMS cryptographic compliance details

Updated the KMS cryptographic compliance documentation to reflect changes in HMAC authentication and added details about the internode RPC HMAC authentication.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Hauser Zhi <housemecn@gmail.com>

---------

Signed-off-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

CHANGELOG.md

@@ -26,7 +26,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - IAM snapshot/deletion replay, target-assigned delete-marker purges, timestamp ordering, and best-effort peer broadcast now close the control-plane gaps found by the R6 review ([#7195](https://github.com/rustfs/rustfs/pull/7195)).
 - A `set-remote-target?update=true&creds=true` request on a site-replication peer's bucket target is now rejected with `400 InvalidRequest`, naming the target and why it counts as peer-owned. It used to return success while keeping the stored endpoint, credentials, and TLS settings, so an endpoint edit from the Console's bucket replication form appeared to apply but did not. Change a peer's address with a site-replication edit instead; see "Changing a site's address" in `docs/operations/site-replication-operations.md`. The `sync`, `bandwidth`, `proxy`, and `path` update groups are unchanged.
 - Site-replication edits that omit the site name, such as `mc admin replicate update --deployment-id <ID> --endpoint <URL>`, now keep the stored name on the editing site and on every receiver instead of renaming the site after the new endpoint host. No persisted format changes.
-- The persisted replication MRF journal is now replayed every 5 minutes while the server runs, not only at startup. Delete-marker purges whose targets were unreachable, and replications that overflowed the worker queues, now converge within one replay interval after the target returns instead of waiting for a restart. A target that stays down is re-attempted once per retained entry per interval. No persisted format changes.
 - Upgrade and rollback: upgrade every node in one site consecutively and verify reconciliation before moving to the next site; do not intentionally run a site mixed-version. Target-version ledger keys are harmless on rollback, although old code cannot use their routing. Before rolling back past [#7307](https://github.com/rustfs/rustfs/pull/7307), drain or repair every pending version purge: older code can free a retained version's data directory before its remote purge is acknowledged. See `docs/operations/site-replication-operations.md`.
 
 ### Security

Cargo.lock

@@ -11058,6 +11058,7 @@ dependencies = [
 name = "rustfs-utils"
 version = "1.0.2"
 dependencies = [
+ "aws-lc-rs",
  "base64-simd",
  "blake2",
  "brotli 9.1.1",

Cargo.toml

@@ -212,6 +212,7 @@ serde_urlencoded = "0.7.1"
 # releases.
 aes-gcm = { version = "0.11.1" }
 argon2 = { version = "0.6.0" }
+aws-lc-rs = { version = "1.18.0", default-features = false }
 blake2 = "0.11.0"
 chacha20poly1305 = { version = "0.11.0" }
 crc-fast = "1.10.0"

crates/ecstore/Cargo.toml

@@ -31,7 +31,7 @@ workspace = true
 
 [features]
 default = []
-fips = ["rustfs-s3-client/fips"]
+fips = ["rustfs-s3-client/fips", "rustfs-utils/fips"]
 diagnostic-read-version-caller = ["rustfs-io-metrics/diagnostic-read-version-caller"]
 # Sampled wall/CPU diagnostics only; does not change the read backend.
 diagnostic-local-batch = ["rustix/time"]

crates/ecstore/src/bucket/replication/replication_pool.rs

@@ -86,10 +86,6 @@ const MRF_CORRUPT_FILE_PREFIX: &str = "config/replication/mrf.corrupt";
 const MRF_PENDING_CAP: usize = 200_000;
 const MRF_RETRY_INITIAL_DELAY: Duration = Duration::from_millis(100);
 const MRF_RETRY_MAX_DELAY: Duration = Duration::from_secs(5);
-/// Interval between runtime replays of the MRF journal. A pass re-attempts
-/// every retained entry, so a target that stays down costs one attempt per
-/// entry per interval.
-const MRF_REPLAY_INTERVAL: Duration = Duration::from_secs(300);
 
 #[derive(Debug, Default)]
 pub struct DurableMrfBacklog {
@@ -619,7 +615,7 @@ async fn read_mrf_entries<S: ReplicationObjectIO>(storage: Arc<S>) -> Result<Vec
 }
 
 /// Acknowledge only the generation read by the recovery leader. Entries appended
-/// after that generation are retained as a suffix and replayed on the next pass.
+/// after that generation are retained as a suffix and replayed on the next startup.
 /// Lock order: recovery leader lock -> MRF journal object lock.
 async fn acknowledge_mrf_recovery<S: ReplicationStorage>(
     storage: Arc<S>,
@@ -671,7 +667,7 @@ async fn acknowledge_mrf_recovery<S: ReplicationStorage>(
     Err(EcstoreError::PreconditionFailed)
 }
 
-/// Acquires the MRF recovery leader lock for one journal replay pass.
+/// Acquires the MRF recovery leader lock for the startup replay.
 /// Returns `None` (after logging) when the lock cannot be created or another
 /// node is already processing the backlog.
 async fn acquire_mrf_recovery_guard<S: ReplicationStorage>(storage: &Arc<S>) -> Option<rustfs_lock::NamespaceLockGuard> {
@@ -1022,7 +1018,7 @@ fn admitted_mrf_replicate_object(oi: ObjectInfo, entry: &MrfReplicateEntry, op_t
 }
 
 /// Acknowledges the replayed MRF prefix and returns the retained backlog.
-/// On acknowledgement failure the backlog is preserved for the next replay pass and
+/// On acknowledgement failure the backlog is preserved for the next startup and
 /// re-read (falling back to the replayed snapshot) so the published summary stays accurate.
 async fn resolve_retained_mrf_entries<S: ReplicationStorage>(
     storage: &Arc<S>,
@@ -1037,7 +1033,7 @@ async fn resolve_retained_mrf_entries<S: ReplicationStorage>(
                 component = LOG_COMPONENT_ECSTORE,
                 subsystem = LOG_SUBSYSTEM_REPLICATION,
                 error = %error,
-                "Failed to acknowledge the MRF recovery prefix; preserving it for the next replay pass"
+                "Failed to acknowledge the MRF recovery prefix; preserving it for the next startup"
             );
             match read_mrf_entries(storage.clone()).await {
                 Ok(current) => current,
@@ -1055,62 +1051,6 @@ async fn resolve_retained_mrf_entries<S: ReplicationStorage>(
     }
 }
 
-/// One MRF journal replay pass. The recovery leader reads the journal,
-/// re-drives every entry, and acknowledges the replayed prefix; entries that
-/// could not be admitted, and entries appended meanwhile, stay for the next pass.
-async fn replay_mrf_journal<S: ReplicationStorage>(storage: &Arc<S>) {
-    let Some(recovery_guard) = acquire_mrf_recovery_guard(storage).await else {
-        return;
-    };
-
-    let Some(entries) = load_mrf_recovery_entries(storage).await else {
-        return;
-    };
-    set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&entries));
-
-    let total = entries.len();
-    let mut queued_count = 0usize;
-    let mut retry_entries = Vec::new();
-
-    for entry in entries.iter() {
-        let Some(admission) = replay_mrf_entry(entry, storage, &mut retry_entries).await else {
-            continue;
-        };
-
-        if admission == ReplicationQueueAdmission::Missed {
-            retry_entries.push(entry.clone());
-        } else if admission == ReplicationQueueAdmission::Queued {
-            queued_count += 1;
-        }
-    }
-
-    let retained = resolve_retained_mrf_entries(storage, &recovery_guard, &entries, &retry_entries).await;
-    let retained_count = retained.len();
-    set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&retained));
-
-    if queued_count > 0 {
-        info!(
-            component = LOG_COMPONENT_ECSTORE,
-            subsystem = LOG_SUBSYSTEM_REPLICATION,
-            recovered = queued_count,
-            total,
-            retained = retained_count,
-            "Replayed MRF entries admitted for retry"
-        );
-    }
-}
-
-/// Re-runs [`replay_mrf_journal`] every `period`. The recovery leader lock keeps
-/// one pass at a time across the cluster, including the startup pass.
-fn spawn_mrf_replay_loop<S: ReplicationStorage>(storage: Arc<S>, period: Duration) -> JoinHandle<()> {
-    tokio::spawn(async move {
-        loop {
-            tokio::time::sleep(period).await;
-            replay_mrf_journal(&storage).await;
-        }
-    })
-}
-
 #[derive(Debug, thiserror::Error)]
 #[error("replication resync {active_resync_id} is already active for {bucket}/{arn}")]
 struct ResyncActiveConflictError {
@@ -1231,7 +1171,6 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
         // Start background tasks
         pool.start_mrf_persister().await;
         pool.start_mrf_processor().await;
-        pool.start_mrf_replayer().await;
         pool.start_force_delete_processor().await;
 
         pool
@@ -1700,21 +1639,52 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
 
     /// Starts the MRF processor — one-shot at startup.
     ///
-    /// Replays the on-disk MRF journal once; [`Self::start_mrf_replayer`] repeats
-    /// the pass while the server runs.
+    /// Reads the on-disk MRF file, re-injects admitted entries as Heal operations, and
+    /// rewrites any entries that could not be admitted for a later startup retry.
     async fn start_mrf_processor(&self) {
         let storage = self.storage.clone();
-        let handle = tokio::spawn(async move { replay_mrf_journal(&storage).await });
-        self.task_handles.lock().await.push(handle);
-    }
 
-    /// Starts the runtime MRF replayer — ongoing background task.
-    ///
-    /// Entries reach the journal while the server runs (worker-queue overflow,
-    /// delete-marker purge intents whose watch window drained with targets still
-    /// dirty). Replaying only at startup left them waiting for a restart.
-    async fn start_mrf_replayer(&self) {
-        let handle = spawn_mrf_replay_loop(self.storage.clone(), MRF_REPLAY_INTERVAL);
+        let handle = tokio::spawn(async move {
+            let Some(recovery_guard) = acquire_mrf_recovery_guard(&storage).await else {
+                return;
+            };
+
+            let Some(entries) = load_mrf_recovery_entries(&storage).await else {
+                return;
+            };
+            set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&entries));
+
+            let total = entries.len();
+            let mut queued_count = 0usize;
+            let mut retry_entries = Vec::new();
+
+            for entry in entries.iter() {
+                let Some(admission) = replay_mrf_entry(entry, &storage, &mut retry_entries).await else {
+                    continue;
+                };
+
+                if admission == ReplicationQueueAdmission::Missed {
+                    retry_entries.push(entry.clone());
+                } else if admission == ReplicationQueueAdmission::Queued {
+                    queued_count += 1;
+                }
+            }
+
+            let retained = resolve_retained_mrf_entries(&storage, &recovery_guard, &entries, &retry_entries).await;
+            let retained_count = retained.len();
+            set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&retained));
+
+            if queued_count > 0 {
+                info!(
+                    component = LOG_COMPONENT_ECSTORE,
+                    subsystem = LOG_SUBSYSTEM_REPLICATION,
+                    recovered = queued_count,
+                    total,
+                    retained = retained_count,
+                    "Replayed MRF entries admitted for retry"
+                );
+            }
+        });
         self.task_handles.lock().await.push(handle);
     }
 
@@ -1785,7 +1755,7 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
     /// writes them to the on-disk MRF file every flush interval (default 10s,
     /// overridable via `RUSTFS_REPL_MRF_FLUSH_INTERVAL_MS`) or when 1 000 new
     /// entries accumulate. Runtime writers append under the journal lock; only the
-    /// recovery leader of a replay pass may remove an acknowledged prefix.
+    /// startup recovery leader may remove an acknowledged prefix.
     async fn start_mrf_persister(&self) {
         let Some(mut rx) = self.mrf_save_rx.lock().await.take() else {
             return;
@@ -6403,43 +6373,6 @@ mod tests {
         assert_eq!(retained.target_arns, entry.target_arns);
     }
 
-    #[tokio::test]
-    async fn mrf_replay_loop_drains_entries_persisted_after_startup() {
-        let shared = empty_resync_shared_state();
-        let storage = Arc::new(LoadResyncNodeStore::new("mrf-runtime-replay", shared.clone()));
-        // A force-delete intent from a superseded generation replays as
-        // Skipped, so a pass that reaches it acknowledges it off the journal.
-        let stale_intent = |object: &str| MrfReplicateEntry {
-            bucket: "mrf-runtime-replay".to_string(),
-            object: object.to_string(),
-            op: MrfOpKind::Delete,
-            force_delete_id: Some(uuid::Uuid::new_v4()),
-            force_delete_local_commit: false,
-            target_arns: vec!["arn:rustfs:replication:target-a".to_string()],
-            ..Default::default()
-        };
-        let journal_drained = || async {
-            tokio::time::timeout(Duration::from_secs(5), async {
-                while !shared.data.lock().expect("test data lock should not be poisoned").is_empty() {
-                    tokio::time::sleep(Duration::from_millis(10)).await;
-                }
-            })
-            .await
-            .is_ok()
-        };
-
-        let handle = spawn_mrf_replay_loop(storage, Duration::from_millis(20));
-        for object in ["first", "second"] {
-            *shared.data.lock().expect("test data lock should not be poisoned") =
-                encode_mrf_file(&[stale_intent(object)]).expect("MRF entry should encode");
-            assert!(
-                journal_drained().await,
-                "an MRF entry persisted while the server runs must be replayed without a restart ({object})"
-            );
-        }
-        handle.abort();
-    }
-
     #[test]
     fn mrf_entry_delete_marker_roundtrip() {
         let dm_vid = Uuid::new_v4();

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -3015,7 +3015,7 @@ const DELETE_MARKER_PURGE_WATCH_INTERVAL: TokioDuration = TokioDuration::from_se
 /// If the marker disappears (deleted before or while the replica landed),
 /// purge the replicated marker from the targets, retrying failed targets on
 /// later rounds. When the window drains with targets still dirty, persist the
-/// purge intent as a durable MRF entry so the next MRF replay pass re-drives it through
+/// purge intent as a durable MRF entry so the next startup replays it through
 /// `purge_stale_delete_marker_targets`.
 async fn watch_and_purge_source_delete_marker<S: ReplicationStorage>(
     bucket: String,
@@ -3106,7 +3106,7 @@ async fn enqueue_delete_marker_purge_mrf(dobj: &DeletedObjectReplicationInfo, fa
 /// Heal callers retry for the full watch window because the startup MRF
 /// processor runs before bucket metadata (and thus target clients) finishes
 /// initializing — the first pass can see `target_client_missing` and a later
-/// round resolves the client; replay passes are serial background work, so
+/// round resolves the client; the replay loop is serial and startup-only, so
 /// blocking it for up to the window per dirty entry is acceptable. Live
 /// callers run on replication workers where a down target would pin a worker
 /// for the whole window, so they attempt once and lean on the durable intent

crates/ecstore/src/cluster/rpc/http_auth.rs

@@ -32,7 +32,6 @@ use crate::storage_api_contracts::internode::{
     PUT_FILE_AUTH_TRAILER_MAGIC, PUT_FILE_CAPABILITY_VERSION,
 };
 
-use hmac::{Hmac, KeyInit, Mac};
 use http::uri::Authority;
 use http::{HeaderMap, HeaderValue, Method, Uri};
 #[cfg(test)]
@@ -47,6 +46,7 @@ use rustfs_io_metrics::internode_metrics::{
     global_internode_metrics,
 };
 use rustfs_object_data_cache::{MemoryBasis, resolve_effective_memory};
+use rustfs_utils::crypto::{HmacSha256, KeyInit, Mac};
 use rustfs_utils::get_env_bool;
 use sha2::Digest as _;
 use sha2::Sha256;
@@ -58,8 +58,6 @@ use time::OffsetDateTime;
 use tracing::{error, info, warn};
 use uuid::Uuid;
 
-type HmacSha256 = Hmac<Sha256>;
-
 const SIGNATURE_HEADER: &str = "x-rustfs-signature";
 pub const RPC_BACKGROUND_ADMISSION_HEADER: &str = "x-rustfs-rpc-background-admission";
 const RPC_BACKGROUND_ADMISSION_V1: &str = "v1";

crates/utils/Cargo.toml

@@ -26,6 +26,7 @@ categories = ["web-programming", "development-tools", "cryptography"]
 
 [dependencies]
 hotpath.workspace = true
+aws-lc-rs = { workspace = true, optional = true }
 base64-simd = { workspace = true, optional = true }
 blake2 = { workspace = true, optional = true }
 brotli = { workspace = true, optional = true }
@@ -89,6 +90,7 @@ path = []  # path manipulation features
 compress = ["dep:flate2", "dep:brotli", "dep:snap", "dep:lz4", "dep:zstd"]
 string = ["dep:regex"]
 crypto = ["dep:base64-simd", "dep:hex-simd", "dep:hmac", "dep:hyper", "dep:sha1", "dep:sha2"]
+fips = ["crypto", "dep:aws-lc-rs", "aws-lc-rs/fips"]
 hash = ["dep:highway", "dep:md-5", "dep:sha2", "dep:blake2", "dep:serde", "dep:siphasher", "dep:hex-simd", "dep:crc-fast"]
 os = ["dep:rustix", "dep:tempfile", "dep:windows"]  # operating system utilities
 integration = []  # integration test features

crates/utils/src/crypto.rs

@@ -13,12 +13,77 @@
 // limitations under the License.
 
 use hex_simd::{AsOut, AsciiCase};
-use hmac::{Hmac, KeyInit, Mac};
+use hmac::Hmac;
+#[cfg(not(feature = "fips"))]
+pub use hmac::{KeyInit, Mac};
 use hyper::body::Bytes;
+#[cfg(not(feature = "fips"))]
 use sha1::Sha1;
 use sha2::{Digest, Sha256};
 use std::mem::MaybeUninit;
 
+#[cfg(not(feature = "fips"))]
+pub type HmacSha256 = Hmac<Sha256>;
+
+#[cfg(feature = "fips")]
+pub struct HmacSha256Output([u8; 32]);
+
+#[cfg(feature = "fips")]
+impl HmacSha256Output {
+    #[must_use]
+    pub fn into_bytes(self) -> hmac::digest::Output<Hmac<Sha256>> {
+        self.0.into()
+    }
+}
+
+#[cfg(feature = "fips")]
+pub trait KeyInit: Sized {
+    fn new_from_slice(key: &[u8]) -> Result<Self, hmac::digest::InvalidLength>;
+}
+
+#[cfg(feature = "fips")]
+pub trait Mac: Sized {
+    fn update(&mut self, data: &[u8]);
+    fn finalize(self) -> HmacSha256Output;
+    fn verify_slice(self, tag: &[u8]) -> Result<(), hmac::digest::MacError>;
+}
+
+#[cfg(feature = "fips")]
+#[derive(Clone)]
+pub struct HmacSha256(aws_lc_rs::hmac::Context);
+
+#[cfg(feature = "fips")]
+impl HmacSha256 {
+    pub fn new_from_slice(key: &[u8]) -> Result<Self, hmac::digest::InvalidLength> {
+        let signing_key = aws_lc_rs::hmac::Key::new(aws_lc_rs::hmac::HMAC_SHA256, key);
+        Ok(Self(aws_lc_rs::hmac::Context::with_key(&signing_key)))
+    }
+}
+
+#[cfg(feature = "fips")]
+impl KeyInit for HmacSha256 {
+    fn new_from_slice(key: &[u8]) -> Result<Self, hmac::digest::InvalidLength> {
+        Self::new_from_slice(key)
+    }
+}
+
+#[cfg(feature = "fips")]
+impl Mac for HmacSha256 {
+    fn update(&mut self, data: &[u8]) {
+        self.0.update(data);
+    }
+
+    fn finalize(self) -> HmacSha256Output {
+        let tag = self.0.sign();
+        HmacSha256Output(tag.as_ref().try_into().expect("HMAC-SHA256 tag is 32 bytes"))
+    }
+
+    fn verify_slice(self, tag: &[u8]) -> Result<(), hmac::digest::MacError> {
+        let computed = self.0.sign();
+        aws_lc_rs::constant_time::verify_slices_are_equal(computed.as_ref(), tag).map_err(|_| hmac::digest::MacError)
+    }
+}
+
 /// Base64 URL safe encoding without padding
 /// `base64_encode_url_safe_no_pad(input)`
 ///
@@ -84,9 +149,18 @@ pub fn is_sha256_checksum(s: &str) -> bool {
 /// A 20-byte array containing the HMAC-SHA1 hash of the input data using the provided key
 ///
 pub fn hmac_sha1(key: impl AsRef<[u8]>, data: impl AsRef<[u8]>) -> [u8; 20] {
-    let mut m = <Hmac<Sha1>>::new_from_slice(key.as_ref()).expect("operation should succeed");
-    m.update(data.as_ref());
-    m.finalize().into_bytes().into()
+    #[cfg(feature = "fips")]
+    {
+        let signing_key = aws_lc_rs::hmac::Key::new(aws_lc_rs::hmac::HMAC_SHA1_FOR_LEGACY_USE_ONLY, key.as_ref());
+        let tag = aws_lc_rs::hmac::sign(&signing_key, data.as_ref());
+        tag.as_ref().try_into().expect("HMAC-SHA1 tag is 20 bytes")
+    }
+    #[cfg(not(feature = "fips"))]
+    {
+        let mut m = <Hmac<Sha1>>::new_from_slice(key.as_ref()).expect("operation should succeed");
+        m.update(data.as_ref());
+        m.finalize().into_bytes().into()
+    }
 }
 
 /// HMAC-SHA256 hashing
@@ -100,7 +174,7 @@ pub fn hmac_sha1(key: impl AsRef<[u8]>, data: impl AsRef<[u8]>) -> [u8; 20] {
 /// A 32-byte array containing the HMAC-SHA256 hash of the input data using the provided key
 ///
 pub fn hmac_sha256(key: impl AsRef<[u8]>, data: impl AsRef<[u8]>) -> [u8; 32] {
-    let mut m = Hmac::<Sha256>::new_from_slice(key.as_ref()).expect("operation should succeed");
+    let mut m = HmacSha256::new_from_slice(key.as_ref()).expect("operation should succeed");
     m.update(data.as_ref());
     m.finalize().into_bytes().into()
 }
@@ -167,3 +241,20 @@ fn sha256_checksum_rejects_wrong_length_or_non_lowercase_hex() {
     assert!(!is_sha256_checksum("0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdeg"));
     assert!(!is_sha256_checksum("0123456789ABCDEF0123456789abcdef0123456789abcdef0123456789abcdef"));
 }
+
+#[test]
+fn test_hmac_sha256_rfc4231() {
+    #[cfg(feature = "fips")]
+    aws_lc_rs::try_fips_mode().expect("aws-lc-rs FIPS mode must be active when `rustfs-utils/fips` is enabled");
+
+    let key = [0x0b_u8; 20];
+    let data = b"Hi There";
+    assert_eq!(hex(hmac_sha1(key, data)), "b617318655057264e28bc0b6fb378c8ef146be00");
+    let mac = hmac_sha256(key, data);
+    assert_eq!(hex(mac), "b0344c61d8db38535ca8afceaf0bf12b881dc200c9833da726e9376c2e32cff7");
+
+    let mut verifier = <HmacSha256 as KeyInit>::new_from_slice(&key).expect("valid key");
+    verifier.update(data);
+    assert!(verifier.clone().verify_slice(&mac).is_ok());
+    assert!(verifier.verify_slice(&[0_u8; 32]).is_err());
+}

docs/operations/kms-cryptographic-compliance.md

@@ -14,6 +14,7 @@ This document records where RustFS stands on cryptographic module validation and
 | Layer | Where | Implementation | Primitives | Validated module |
 | --- | --- | --- | --- | --- |
 | TLS using the process-wide rustls provider | Provider installed or checked by `install_default_crypto_provider` in `rustfs/src/startup_runtime_hooks.rs` | `rustls` with the process-wide provider (AWS-LC is installed when none exists) | TLS 1.2/1.3 suites, `prefer-post-quantum` hybrid key exchange (restricted to FIPS-approved suites and groups when `--features fips` is enabled) | No in the default build (`aws-lc-sys`); opt-in `--features fips` enables `aws-lc-fips-sys` and checks the installed provider at startup; module validation depends on the deployment |
+| Internode RPC HMAC authentication | `crates/ecstore/src/cluster/rpc/http_auth.rs` via `rustfs_utils::crypto::HmacSha256` | RustCrypto `hmac` + `sha2` in the default build; `aws-lc-rs` when `fips` is enabled | HMAC-SHA256 (`x-rustfs-signature`, `x-rustfs-rpc-signature-v2`, `x-rustfs-rpc-signature-v3`) | No in the default build; opt-in `--features fips` enables `rustfs-utils/fips` (`aws-lc-fips-sys`) |
 | Object data path AEAD (SSE) | `crates/kms/src/encryption/ciphers.rs`, `crates/rio/src/encrypt_reader.rs`, `crates/rio-v2/src/encrypt_reader.rs` | RustCrypto `aes-gcm`, `chacha20poly1305` | AES-256-GCM, ChaCha20-Poly1305 | No |
 | DEK wrapping | `crates/kms/src/encryption/dek.rs` | RustCrypto `aes-gcm` | AES-256-GCM | No |
 | Local KMS backend master key | `crates/kms/src/backends/local.rs` | RustCrypto `argon2`, `aes-gcm` | Argon2id KDF, AES-256-GCM | No |

docs/operations/site-replication-operations.md

@@ -131,10 +131,7 @@ persist until the refresh settles.
 Control-plane retry begins on the 30-second drain, while heavyweight snapshots,
 pending topology operations, and bucket wiring are revisited on the 600-second
 pass. Object MRF entries are persisted every 10 seconds by default and target
-health is probed every 5 seconds. The persisted MRF journal is replayed at
-startup and then every 5 minutes, so a delete-marker purge or an overflowed
-replication whose target was down converges within one replay interval after
-the target returns. These are scheduling bounds, not delivery
+health is probed every 5 seconds. These are scheduling bounds, not delivery
 SLAs: network timeouts and the amount of queued work add to them.
 
 Objects that must be rediscovered by the scanner have this conservative upper

rustfs/Cargo.toml

@@ -58,7 +58,7 @@ required-features = ["swift"]
 
 [features]
 default = ["ftps", "webdav", "gcs"]
-fips = ["rustls/fips", "tokio-rustls/fips", "rustfs-tls-runtime/fips", "rustfs-ecstore/fips", "rustfs-protocols/fips", "rustfs-s3-client/fips", "rustfs-targets/fips"]
+fips = ["rustls/fips", "tokio-rustls/fips", "rustfs-tls-runtime/fips", "rustfs-ecstore/fips", "rustfs-protocols/fips", "rustfs-s3-client/fips", "rustfs-targets/fips", "rustfs-utils/fips"]
 gcs = ["rustfs-ecstore/gcs", "dep:google-cloud-auth"]
 metrics-gpu = ["rustfs-obs/gpu"]
 ftps = ["rustfs-protocols/ftps"]