fix(targets): harden control-plane, TLS reload, and runtime extension points (#4504)

Addresses security/correctness audit findings in the target-plugin control
plane, TLS reload coordinator, and runtime extension registries.

control_plane (backlog#977):
- Install now preserves the currently installed revision as previous_revision
  so Rollback actually restores the prior version instead of a no-op.
- Split the gate: circuit-breaker and runtime-activation checks apply only to
  Install/Enable; Disable and Rollback stay available as break-glass
  remediation while the breaker is open.
- Enforce the sidecar runtime protocol version for every external transport
  (previously skippable by declaring a non-gRPC transport) and validate the
  plugin api_compatibility_version at planning time.
- Download/signature/provenance host allowlisting now matches the full
  host authority, so an allowlisted host never implicitly authorizes a
  different host:port.

TLS reload coordinator/validate (backlog#981, #970-coordinator):
- Compute the fingerprint before building material in register() to remove the
  TOCTOU that could permanently pin an old certificate; rotation self-heals.
- Always start a detection loop when reload is enabled; Watch mode no longer
  returns success without any loop.
- validate_cert_key_pairing now verifies the private key matches the
  certificate's public key instead of only parsing both files.
- Normalize a zero poll interval to a positive minimum to avoid a panic that
  silently killed the poll loop.
- Serialize reload cycles with a per-target mutex; a first-step fingerprint
  read failure now records last_error and a failure metric.
- Duplicate label registration stops and joins the previous loop before
  publishing the replacement (stop-before-start), preventing orphaned loops.

runtime extension points (backlog#983 runtime subset):
- ops_profiler/ops_diagnostics authorize before probing the registry so an
  unauthorized caller cannot learn whether a backend/surface exists.
- s3_hooks dispatch_post_auth actually traverses the registered hooks for the
  point instead of unconditionally returning Continue.
- sidecar send_with_timeout redacts errors and uses the configurable failure
  threshold via policy, and a successful send resets the breaker accounting.

Relates to rustfs/backlog#977
Relates to rustfs/backlog#981
Relates to rustfs/backlog#970
Relates to rustfs/backlog#983

Co-authored-by: heihutu <heihutu@gmail.com>

crates/targets/src/runtime/sidecar.rs

@@ -187,19 +187,32 @@ impl SidecarPluginRuntime {
         }
     }
 
-    pub fn send_with_timeout(&mut self, operation_timeout: Duration, simulated_latency: Duration) -> Result<(), String> {
-        if simulated_latency > operation_timeout {
-            self.record_failure(format!(
-                "sidecar send timeout after {:?} (budget {:?})",
-                simulated_latency, operation_timeout
-            ));
+    /// Simulates a send bounded by the policy's operation timeout. On timeout the
+    /// failure is recorded through the policy (so error details are redacted when
+    /// `redact_error_details` is set and the configurable failure threshold —
+    /// not a hardcoded constant — drives circuit breaking). A successful send
+    /// clears the failure count so transient blips never accumulate toward the
+    /// breaker.
+    pub fn send_with_timeout(&mut self, policy: &SidecarRuntimePolicy, simulated_latency: Duration) -> Result<(), String> {
+        if simulated_latency > policy.operation_timeout {
+            self.record_failure_with_policy(
+                policy,
+                format!(
+                    "sidecar send timeout after {:?} (budget {:?})",
+                    simulated_latency, policy.operation_timeout
+                ),
+            );
             return Err(self
                 .last_error
                 .clone()
-                .unwrap_or_else(|| "sidecar timeout without recorded error".to_string()));
+                .unwrap_or_else(|| "sidecar operation failed".to_string()));
         }
         self.healthy = true;
         self.last_error = None;
+        // Success resets the circuit-breaker accounting: a healthy send must not
+        // leave stale failures that could trip the breaker on the next blip.
+        self.failure_count = 0;
+        self.degraded_to_builtin = false;
         Ok(())
     }
 
@@ -404,12 +417,42 @@ mod tests {
     }
 
     #[test]
-    fn sidecar_runtime_send_timeout_records_last_error() {
+    fn sidecar_runtime_send_timeout_redacts_error_and_uses_policy_threshold() {
         let mut runtime = SidecarPluginRuntime::new("grpc://127.0.0.1:50051", notify_sidecar_handshake());
+        // redact_error_details defaults to true; threshold 2 is honored, not the
+        // hardcoded DEFAULT_FAILURE_THRESHOLD.
+        let policy = SidecarRuntimePolicy::verified_external(16, Duration::from_millis(50), 2);
 
-        let result = runtime.send_with_timeout(Duration::from_millis(50), Duration::from_millis(75));
+        let result = runtime.send_with_timeout(&policy, Duration::from_millis(75));
 
         assert!(result.is_err());
-        assert_eq!(runtime.last_error.as_deref(), Some("sidecar send timeout after 75ms (budget 50ms)"));
+        // The raw budget/latency detail must not leak; it is redacted.
+        assert_eq!(runtime.last_error.as_deref(), Some("sidecar operation failed"));
+        assert_eq!(runtime.failure_count, 1);
+        assert!(!runtime.degraded_to_builtin);
+
+        // Second timeout hits the configured threshold and degrades.
+        let _ = runtime.send_with_timeout(&policy, Duration::from_millis(75));
+        assert_eq!(runtime.failure_count, 2);
+        assert!(runtime.degraded_to_builtin);
+    }
+
+    #[test]
+    fn sidecar_runtime_successful_send_clears_failure_count() {
+        let mut runtime = SidecarPluginRuntime::new("grpc://127.0.0.1:50051", notify_sidecar_handshake());
+        let policy = SidecarRuntimePolicy::verified_external(16, Duration::from_millis(50), 3);
+
+        // Accumulate a failure, then a successful send must reset the breaker
+        // accounting so a later single failure does not immediately degrade.
+        let _ = runtime.send_with_timeout(&policy, Duration::from_millis(75));
+        assert_eq!(runtime.failure_count, 1);
+
+        runtime
+            .send_with_timeout(&policy, Duration::from_millis(10))
+            .expect("a within-budget send should succeed");
+        assert_eq!(runtime.failure_count, 0);
+        assert!(!runtime.degraded_to_builtin);
+        assert!(runtime.healthy);
+        assert!(runtime.last_error.is_none());
     }
 }