fix(ecstore): send valid ping body in remote locker (#3112)

* fix(ecstore): send valid ping body in remote locker

Build ping requests with a flatbuffer payload so health checks remain compatible with the ping response parser after restart.

* fix(bench): use multi-host warp target during failover

Normalize comma-separated warp host lists in run_object_batch_bench and let four-node failover bench pass BENCH_WARP_HOSTS so rolling restart does not pin load to a single restarting node.

* feat(health): add compat health probes with busy/KMS checks

  - Add /health/live liveness probe endpoint
  - Add busy protection (429) for readiness probes, gated by RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE
  - Add KMS readiness check for /health/ready, gated by RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE
  - Add lock quorum status caching with TTL to reduce RPC pressure
  - Consolidate health response building into build_health_response_parts
  - Register /health/live in console router and readiness gate
  - Remove MinIO references from newly added health code

* fix(health): decouple kms readiness from lock quorum
This commit is contained in:
houseme
2026-05-29 16:02:50 +08:00
committed by GitHub
parent c257043b63
commit 2b82432f9e
14 changed files with 578 additions and 169 deletions
+17
View File
@@ -26,3 +26,20 @@ pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
/// When enabled, only `status` and `ready` fields are returned.
pub const ENV_HEALTH_MINIMAL_RESPONSE_ENABLE: &str = "RUSTFS_HEALTH_MINIMAL_RESPONSE_ENABLE";
pub const DEFAULT_HEALTH_MINIMAL_RESPONSE_ENABLE: bool = false;
/// Enable busy protection for health probes.
/// When enabled with a positive request threshold, alias health probes may
/// return 429 when active HTTP requests exceed the threshold.
pub const ENV_HEALTH_COMPAT_BUSY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE";
pub const DEFAULT_HEALTH_COMPAT_BUSY_CHECK_ENABLE: bool = false;
/// Max active HTTP requests; alias health probes report busy (429) when active requests reach or exceed this value.
/// Set to 0 to disable thresholding even when busy protection is enabled.
pub const ENV_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS: &str = "RUSTFS_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS";
pub const DEFAULT_HEALTH_COMPAT_BUSY_MAX_ACTIVE_REQUESTS: usize = 0;
/// Enable KMS readiness check for alias readiness probes.
/// When enabled, `/health/ready` additionally requires KMS service to be
/// in running state if a global KMS manager exists.
pub const ENV_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE: &str = "RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE";
pub const DEFAULT_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE: bool = false;