3 Commits

Author	SHA1	Message	Date
Zhengchao An	35e4415ed9	feat(kms): expose key lifecycle and Vault credential gauges (#5542) ... * feat(kms): observe key lifecycle from the deletion sweep The sweep already pages through the whole key set, so the lifecycle gauges come out of the pages it has in hand: no extra backend call is made for them. It publishes the number of keys awaiting their deletion deadline, the number of tombstones an interrupted removal left behind, and how long ago the least recently rotated usable key was rotated (counting from creation for keys that were never rotated), plus a per-outcome counter of what the sweep acted on. Every gauge is a label-less aggregate: a per-key label would carry key identifiers into the metric stream and grow the series count with the key set, so "this key is overdue for rotation" stays a threshold for an alerting rule to apply to the aggregate. Keys the sweep destroys drop out of the census, and a sweep that could not finish listing leaves the gauges at their last complete values rather than understating them. * feat(kms): expose Vault token TTL and fail-closed state as gauges The renewal loop already tracks token expiry, so it now publishes the seconds left on the active Vault token and whether the provider is refusing to serve it. The fail-closed gauge re-evaluates the very gate `VaultCredentialProvider::current` applies, so what operators see and what the request path does cannot drift apart. Both waits in the loop republish on a bounded cadence, so a scrape landing between refresh cycles never reads a TTL frozen at the last refresh or a fail-closed state that flipped after it. That costs a timer and no Vault traffic, and the request path stays free of metric work. Renewal successes and failures already land in the auth operation counters, so nothing is double-counted here. Neither gauge carries a label: the address, mount, auth path and token are all off limits as label values, and there is one generation to describe.	2026-08-01 13:52:43 +08:00
Zhengchao An	3921336b23	feat(kms): AppRole login with background token renewal and fail-closed expiry (#5487) ... * feat(kms): add AppRole configuration surface for Vault auth Extend VaultAuthMethod::AppRole with secret_id_file (re-read on every login so external rotation is picked up), a configurable auth mount (default "approle"), and an optional fail-closed safety window. All new fields are serde(default) so previously persisted configurations keep deserializing, and the strict admin-configure deserializer accepts them as optional. Environment selection: setting RUSTFS_KMS_VAULT_APPROLE_ROLE_ID switches both Vault backends to AppRole; the secret_id comes from RUSTFS_KMS_VAULT_APPROLE_SECRET_ID_FILE (path stored, file wins) or RUSTFS_KMS_VAULT_APPROLE_SECRET_ID, following the static secret-key file precedent. validate() rejects AppRole configs without a role_id, without any secret_id source, or with an empty mount. Also append the CredentialsUnavailable error variant used by the fail-closed credential gate. * feat(kms): implement AppRole login with background renewal and fail-closed expiry Implement the AppRoleLogin token source (vaultrs approle login + renew-self) and wire lease-bound credentials through the provider: - Each successful login/renewal installs a new client generation in the ArcSwap; in-flight requests finish on the generation they captured. - A background renewal task refreshes at half the lease TTL: renewable tokens are renewed in place, everything else (or a failed renewal) falls back to a fresh login. Auth exchanges run under the typed retry policy (OpClass::Auth) and failed cycles retry on a fixed cadence, so the provider recovers once Vault does. - Fail-closed: current() refuses to hand out a token inside the configured safety window of its expiry (default: one attempt timeout), returning CredentialsUnavailable instead of sending a request whose token may lapse mid-flight. - Refreshes are single-flight: concurrent triggers for the same generation coalesce into one login. - The renewal task's owner handle lives on the KMS service version: stop() shuts it down explicitly and reconfigure recycles it via cancel-on-drop when the old version is discarded. - The secret_id file is re-read on every login attempt; missing or empty files fail the attempt without contacting Vault. Crate-owned copies of tokens and secret_ids are zeroized on drop, and Debug output of every credential-carrying type stays redacted (leak regression tests). The renewal machinery is covered by paused-clock tests driving a scripted token source: renew-at-half-TTL timing, login fallback, fail-closed window entry and recovery, prompt task recycling, and coalesced concurrent refreshes. * feat(kms): add Vault Agent token file authentication Add the TokenFile source: the token is read from an agent-managed sink file (RUSTFS_KMS_VAULT_TOKEN_FILE or the TokenFile auth config) and re-read once per poll interval (default 30s) through the existing renewal loop, so a token rotated by the agent installs a new client generation within one poll of the atomic replace. Each successful read extends the token's observed validity to twice the poll interval; a file that disappears or turns empty keeps failing the refresh until the fail-closed window trips, and heals the provider as soon as it is restored. Reads are strict and never contact Vault on failure: the file must be non-empty after trimming, and on Unix group/other permission bits are a hard error (mirroring the SFTP host-key rule). Rotation detection uses a content digest; the token itself is never stored on the source and the crate-owned copy is zeroized. Configuring the token file together with AppRole or an explicit static token is rejected as a configuration error. All new config fields are serde(default) and the strict admin-configure deserializer accepts the new variant. Covered by paused-clock tests (atomic replacement installs a new generation next cycle, deletion fails closed and recovers, prompt task recycling) plus negatives for missing/empty/over-permissive files and a Debug leak regression. * docs(kms): add Vault authentication and credential lifecycle runbook Cover choosing between static token, AppRole, and Vault Agent token file auth; AppRole role setup with SecretID delivery and rotation; Agent sink deployment with the permission requirements; and the fail-closed window semantics with a troubleshooting table keyed on the renewal task's log lines.	2026-07-30 22:29:49 +00:00
Zhengchao An	8368017fb2	refactor(kms): route Vault clients through a rotatable credential provider (#5481) ... * fix(kms): repair vault test call sites missed by the timeout refactor Three offline tests still constructed VaultKmsClient with the pre-#5472 single-argument signature, leaving cargo test -p rustfs-kms unable to compile. Pass the same 30s attempt timeout the neighbouring tests use. * refactor(kms): route Vault clients through a rotatable credential provider Both Vault backends previously built a VaultClient in their constructor and held it for the lifetime of the backend, which leaves no seam for re-authentication: rotating credentials would require tearing down the whole backend. Introduce backends/vault_credentials with a TokenSource trait (only StaticToken for now; AppRole login and agent token files land in follow-ups) and a VaultCredentialProvider that owns the authenticated client behind an ArcSwap. Request paths take a per-call snapshot via current(), so a future rotation swaps in a new client generation without interrupting calls already in flight. Tokens held by this crate are zeroized on drop, and Debug output of every credential-carrying type is redacted (covered by a leak regression test). Behavior is unchanged: static token, namespace, and per-attempt timeout feed the same VaultClientSettings as before, and AppRole configurations are still rejected at construction with the same message.	2026-07-30 16:48:24 +00:00