gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-03 03:47:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(kms): persist and report the Vault KV2 key rotation timestamp (#5594)

Browse Source 
* fix(kms): persist and report the Vault KV2 key rotation timestamp

The rotation-age gauge reads KeyInfo::rotated_at and falls back to
created_at when it is absent. The KV2 backend never persisted a rotation
time and hardcoded None in describe_key, so a key rotated many times and
a key that was never rotated reported the same age.

Record the rotation time on the same check-and-set write that switches
the current version, and report the stored value from describe_key and
from the recovered-create path. Records written before the field existed
keep deserializing and stay unstamped: no timestamp is invented for a
rotation this node cannot vouch for.

* test(kms): cover Vault KV2 rotation timestamp persistence and legacy records
This commit is contained in:
Zhengchao An
2026-08-02 05:31:51 +08:00
committed by GitHub
parent da6fc5314d
commit a29ae4e5cd
3 changed files with 178 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/operations/kms-observability-runbook.md
+3 -1
View File
@@ -48,13 +48,15 @@ Published by the background deletion worker (`crates/kms/src/deletion_worker.rs`
| --- | --- | --- | --- |
| `rustfs_kms_pending_deletion_keys` | gauge | — | Keys scheduled for deletion whose deadline has not passed |
| `rustfs_kms_deletion_tombstone_keys` | gauge | — | Keys left tombstoned by an interrupted removal, still awaiting the sweep |
| `rustfs_kms_oldest_key_rotation_age_seconds` | gauge | — | Seconds since the least recently rotated usable key was rotated, counting from creation for keys never rotated; `0` when there are none |
| `rustfs_kms_oldest_key_rotation_age_seconds` | gauge | — | Seconds since the least recently rotated usable key was rotated, counting from creation for keys with no recorded rotation; `0` when there are none |
| `rustfs_kms_deletion_sweep_keys_total` | counter | `outcome` | Keys the sweep acted on, by outcome |
`outcome` is `removed`, `blocked` (live configuration — the default key, or a reference reported by the injected checker — still points at the key, so the sweep refuses to remove it), `skipped` (pending but not yet due, or the state changed between inspection and removal), or `failed` (the removal attempt failed and is retried next sweep). Every series is emitted at zero from the first sweep on, so a `rate()` over it is defined immediately.
The three gauges are republished only by a sweep that saw the whole key set; a sweep that could not finish listing leaves the previous, complete values standing rather than understating them. Keys already on their way out are excluded from the rotation-age gauge, so it does not stay pinned high by a key that will never be rotated again.
The rotation age comes from whatever the backend reports as the last rotation, and backends only report a rotation they recorded themselves. A key rotated before its backend persisted rotation timestamps therefore ages from creation until its next rotation stamps the record: the gauge overstates that key's age rather than inventing a rotation it cannot vouch for, so an alert on it fires early rather than late. Backends that cannot rotate at all (Local, Static) age every key from creation by construction.
### Vault credential metrics
Published by the Vault credential provider (`crates/kms/src/backends/vault_credentials.rs`), so they exist only on Vault-backed backends. Both are label-less: there is exactly one credential generation to describe, and the Vault address, mount, auth path and token are all off limits as label values.