mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-02 03:19:19 +00:00
Zhengchao An
dd11145a26
* feat(kms): record operation metrics in the retry policy engine Instrument policy::execute — the single choke point every outbound Vault call and credential exchange already flows through — so no call site needs its own instrumentation: - rustfs_kms_backend_operations_total (counter): operation, op_class, outcome (success / fatal / budget_exhausted / deadline_exceeded / cancelled) - rustfs_kms_backend_attempt_failures_total (counter): operation, error_class (retryable_conn / retryable_status / fatal / attempt_timeout) - rustfs_kms_backend_operation_duration_seconds (histogram): wall-clock duration including retries and backoff - rustfs_kms_backend_operation_attempts (histogram): attempts used Metric labels carry only static enum values (operation names, classes, outcomes) — never key identifiers, key material, ciphertext, or tokens. Emission goes through the process-global metrics facade recorder, the same pattern the rest of the workspace uses, so no new wiring is needed in rustfs/src. Tests drive a paused-clock runtime under a thread-local debugging recorder, so counts, attempts, and even the recorded (virtual-clock) durations are asserted deterministically with zero real sleeps. Refs rustfs/backlog#1569 (part of rustfs/backlog#1562) * test(kms): add Vault fault-injection matrix Offline cases inject transport faults locally and are fully deterministic: a refused connection is retried up to the configured budget, and a stalled connection is cut off by the per-attempt timeout instead of hanging. Ignored cases run against a real dev Vault (RUSTFS_KMS_VAULT_ADDR) and pin the fail-closed auth behavior: an invalid token and a missing key each resolve in exactly one attempt. Every case asserts through the policy metrics recorded by a thread-local debugging recorder, which doubles as the request-count assertion even against a real server. Throttling and recoverable 5xx responses cannot be forced on a stock dev Vault; those paths stay pinned by the scripted-Vault wiring tests and the engine tests. Refs rustfs/backlog#1569 (part of rustfs/backlog#1562)