feat(kms): AppRole login with background token renewal and fail-closed expiry (#5487)

* feat(kms): add AppRole configuration surface for Vault auth

Extend VaultAuthMethod::AppRole with secret_id_file (re-read on every
login so external rotation is picked up), a configurable auth mount
(default "approle"), and an optional fail-closed safety window. All new
fields are serde(default) so previously persisted configurations keep
deserializing, and the strict admin-configure deserializer accepts them
as optional.

Environment selection: setting RUSTFS_KMS_VAULT_APPROLE_ROLE_ID switches
both Vault backends to AppRole; the secret_id comes from
RUSTFS_KMS_VAULT_APPROLE_SECRET_ID_FILE (path stored, file wins) or
RUSTFS_KMS_VAULT_APPROLE_SECRET_ID, following the static secret-key file
precedent. validate() rejects AppRole configs without a role_id, without
any secret_id source, or with an empty mount.

Also append the CredentialsUnavailable error variant used by the
fail-closed credential gate.

* feat(kms): implement AppRole login with background renewal and fail-closed expiry

Implement the AppRoleLogin token source (vaultrs approle login +
renew-self) and wire lease-bound credentials through the provider:

- Each successful login/renewal installs a new client generation in the
  ArcSwap; in-flight requests finish on the generation they captured.
- A background renewal task refreshes at half the lease TTL: renewable
  tokens are renewed in place, everything else (or a failed renewal)
  falls back to a fresh login. Auth exchanges run under the typed retry
  policy (OpClass::Auth) and failed cycles retry on a fixed cadence, so
  the provider recovers once Vault does.
- Fail-closed: current() refuses to hand out a token inside the
  configured safety window of its expiry (default: one attempt timeout),
  returning CredentialsUnavailable instead of sending a request whose
  token may lapse mid-flight.
- Refreshes are single-flight: concurrent triggers for the same
  generation coalesce into one login.
- The renewal task's owner handle lives on the KMS service version:
  stop() shuts it down explicitly and reconfigure recycles it via
  cancel-on-drop when the old version is discarded.
- The secret_id file is re-read on every login attempt; missing or empty
  files fail the attempt without contacting Vault. Crate-owned copies of
  tokens and secret_ids are zeroized on drop, and Debug output of every
  credential-carrying type stays redacted (leak regression tests).

The renewal machinery is covered by paused-clock tests driving a
scripted token source: renew-at-half-TTL timing, login fallback,
fail-closed window entry and recovery, prompt task recycling, and
coalesced concurrent refreshes.

* feat(kms): add Vault Agent token file authentication

Add the TokenFile source: the token is read from an agent-managed sink
file (RUSTFS_KMS_VAULT_TOKEN_FILE or the TokenFile auth config) and
re-read once per poll interval (default 30s) through the existing
renewal loop, so a token rotated by the agent installs a new client
generation within one poll of the atomic replace. Each successful read
extends the token's observed validity to twice the poll interval; a
file that disappears or turns empty keeps failing the refresh until the
fail-closed window trips, and heals the provider as soon as it is
restored.

Reads are strict and never contact Vault on failure: the file must be
non-empty after trimming, and on Unix group/other permission bits are a
hard error (mirroring the SFTP host-key rule). Rotation detection uses
a content digest; the token itself is never stored on the source and
the crate-owned copy is zeroized.

Configuring the token file together with AppRole or an explicit static
token is rejected as a configuration error. All new config fields are
serde(default) and the strict admin-configure deserializer accepts the
new variant.

Covered by paused-clock tests (atomic replacement installs a new
generation next cycle, deletion fails closed and recovers, prompt task
recycling) plus negatives for missing/empty/over-permissive files and a
Debug leak regression.

* docs(kms): add Vault authentication and credential lifecycle runbook

Cover choosing between static token, AppRole, and Vault Agent token
file auth; AppRole role setup with SecretID delivery and rotation;
Agent sink deployment with the permission requirements; and the
fail-closed window semantics with a troubleshooting table keyed on the
renewal task's log lines.

crates/kms/src/service_manager.rs

@@ -14,6 +14,7 @@
 
 //! KMS service manager for dynamic configuration and runtime management
 
+use crate::backends::vault_credentials::CredentialTaskHandle;
 use crate::backends::{KmsBackend, local::LocalKmsBackend};
 use crate::config::{BackendConfig, KmsConfig};
 use crate::error::{KmsError, Result};
@@ -110,6 +111,10 @@ struct ServiceVersion {
     service: Arc<ObjectEncryptionService>,
     /// The KMS manager instance
     manager: Arc<KmsManager>,
+    /// Owner of the backend's credential renewal task, if the backend needs
+    /// one. Stop shuts it down explicitly; reconfigure recycles it through
+    /// the handle's cancel-on-drop behavior when the old version is discarded.
+    credential_task: Option<Arc<CredentialTaskHandle>>,
 }
 
 #[derive(Clone)]
@@ -331,6 +336,13 @@ impl KmsServiceManager {
             current_service: None,
         }));
 
+        // Shut down the stopped version's credential renewal task before
+        // reporting stopped, so stop deterministically recycles the background
+        // task even while in-flight operations still hold the old service Arc.
+        if let Some(task) = state.current_service.as_ref().and_then(|sv| sv.credential_task.clone()) {
+            task.shutdown().await;
+        }
+
         debug!(
             event = EVENT_KMS_SERVICE_STATE,
             component = LOG_COMPONENT_KMS,
@@ -488,7 +500,10 @@ impl KmsServiceManager {
 
         info!("Creating KMS service version {} with backend: {:?}", version, config.backend);
 
-        // Create backend
+        // Create backend. Vault backends may also spawn a background
+        // credential renewal task whose owner handle lives on the service
+        // version, so replacing the version recycles the task.
+        let mut credential_task = None;
         let backend = match &config.backend_config {
             BackendConfig::Local(_) => {
                 info!("Creating Local KMS backend for version {}", version);
@@ -498,11 +513,13 @@ impl KmsServiceManager {
             BackendConfig::VaultKv2(_) => {
                 info!("Creating Vault KV2 KMS backend for version {}", version);
                 let backend = crate::backends::vault::VaultKmsBackend::new(config.clone()).await?;
+                credential_task = backend.spawn_credential_renewal().map(Arc::new);
                 Arc::new(backend) as Arc<dyn KmsBackend>
             }
             BackendConfig::VaultTransit(_) => {
                 info!("Creating Vault Transit KMS backend for version {}", version);
                 let backend = crate::backends::vault_transit::VaultTransitKmsBackend::new(config.clone()).await?;
+                credential_task = backend.spawn_credential_renewal().map(Arc::new);
                 Arc::new(backend) as Arc<dyn KmsBackend>
             }
             BackendConfig::Static(_) => {
@@ -522,6 +539,7 @@ impl KmsServiceManager {
             version,
             service: encryption_service,
             manager: kms_manager,
+            credential_task,
         })
     }